RL for Language Models

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

36 papers in the last four weeks, up 71% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 271

All topics

QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides
Jul 17, 2026Zhengyang Zhuge, Hao Yu, Xin Wang +4LLM Quantization RL for Language Models
Qwen Team, Alibaba Inc.
Meta-Learning Preferences for Multilingual LLM Alignment
Jul 14, 2026Jiaying Lin, Seongho Son, Nam Phuong Tran +3RL for Language Models LLM Alignment
Department of Computer Science, University of Warwick, Coventry, United Kingdom · Department of Computer Science, University College London, London, United Kingdom · Department of Mathematics and Computer Science, University of Basel, Basel, Switzerland
From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation
Jul 14, 2026Mehak Dhaliwal, Rasta Tadayon, Andong Hua +2RL for Language Models Confidence Estimation in Language Models
University of California, Santa Barbara
Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
Jul 14, 2026Xinyu Tang, Qianggang Cao, Yurou Liu +13RL for Language Models RL for Language Model Reasoning
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Ant Group · 3Tsinghua University +1
MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignment
Jul 13, 2026Junyoung Park, Namgyu Park, Sechan Lee +3RL for Language Models LLM Jailbreak Attacks
POSTECH GSAI · Samsung SDS · POSTECH GSAI & CSE
Predictive Divergence Masks for LLM RL
Jul 12, 2026Xiangxin Zhou, Jiarui Yao, Penghui Qi +4RL for Language Models Reinforcement Learning
1Tencent Hunyuan · 2UIUC · 3NUS
GAE: Graph-Augmented Evolution for Scientific Discovery via Reinforcement Optimization
Jul 11, 2026Xuanzhou Chen, Taoli ChengRL for Language Models Genetic Programming
Shanghai Academy of AI for Science, Shanghai, China · School of Electrical and Computer Engineering, Georgia Institute of Technology, Georgia, Unites States.
Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation
Jul 9, 2026Ingrid Petrova, Luan VejsiuLLM Self-Correction RL for Language Models
European University of Tirana
When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning
Jul 8, 2026Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang +4RL for Language Models Token-Level Credit Assignment
Johns Hopkins University · Rice University · Workato +1
R^3: Advertisement Compliance Rectification via Group-Relative Experience Extractor and Curriculum Reinforcement
Jul 8, 2026Yuan Chen, Zhenyu Hu, Mengge Xue +5RL for Language Models Video Content Moderation
Tencent
Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design
Jul 7, 2026Alexander Rombach, Chantale Lauer, Nijat MehdiyevRL for Language Models Multi-Objective Language Model Alignment
German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Saarland, Germany · Saarland University, Campus D3 2, Saarbrücken, 66123, Saarland, Germany
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
Jul 7, 2026Chenyu ZhouReward Hacking RL for Language Models
School of Engineering, Institute of Science Tokyo
CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs
Jul 6, 2026Qiuyi Qi, Jinjian Zhang, Mutian Bao +9RL for Language Models LLM Planning
Zhejiang University · Ant Group · City University of Hong Kong
LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
Jul 5, 2026Yujin Kim, Namgyu Ho, Sangmin Hwang +7RL for Language Models Reinforcement Learning
1KAIST · 2Upstage · University of Toronto +2
Aligning Language Models with Selective Prediction
Jul 3, 2026Gaoxiang Luo, Yifan Wu, Sinian Zhang +2RL for Language Models LLM Alignment
Department of Computer Science and Engineering · 2Bioinformatics and Computational Biology Program · 3Division of Biostatistics and Health Data Science
Amortising Bayesian Experimental Design for Sequential Information Gathering in LLMs
Jul 3, 2026Jakob Hartmann, James Harvey, Jhonathan Navott +5RL for Language Models Amortized Inference
University of Oxford, Oxford, United Kingdom · 2Ellison Institute of Technology, Oxford, United Kingdom
Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG
Jul 3, 2026Haotian Zhou, Weiran Huang, Siqi Liu +3RL for Language Models Retrieval-Augmented Generation
School of Computer Science, Shanghai Jiao Tong University · Ant International, Ant Group · Shanghai Innovation Institute +1
Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
Jul 3, 2026Jiyang Guan, Yong Xie, Jun Chen +6RL for Language Models LLM Safety Alignment
Alibaba AAIG
Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
Jul 2, 2026Xuqing Yang, Yi Yuan, Shanzhe Lei +1LLM Inference Efficiency RL for Language Models
Shanghai Jiao Tong University · Southeast University · Shanghai AI Laboratory
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Jul 1, 2026Zijian Zhang, Rizhen Hu, Athanasios Glentis +4RL for Language Models Fine-Tuning
University of Minnesota · Peking University · Amazon
Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations
Jul 1, 2026Mehul Damani, Isha Puri, Idan Shenfeld +1Reward Hacking RL for Language Models
MIT EECS
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
Jun 30, 2026Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona +2RL for Language Models Reinforcement Learning
Yale University · Google Research
GR2 Technical Report
Jun 30, 2026Yufei Li, Zaiwei Zhang, Mingfu Liang +68RL for Language Models LLM Reranking
Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
Jun 29, 2026Ved Sriraman, Peihan Liu, Daniel Hsu +1RL for Language Models Imitation Learning
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning
Jun 25, 2026Ting Zhou, Zhenqing Ling, Yiyang Zhao +2RL for Language Models LLM Alignment
Sun Yat-Sen University, Guangzhou, China · Alibaba Group, Hangzhou, China · Guangdong Tobacco Guangzhou Co., Ltd., Guangzhou, China
NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research
Jun 25, 2026Qiaobo Hao, Yangqian Wu, Shunyi Wang +5Supervised Fine-Tuning RL for Language Models
ZTE NebulaL0 Post-Training Team
ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning
Jun 23, 2026Wenyang Hu, Junxiang Jia, Zhen Shu +3RL for Language Models RL for Language Model Reasoning
1National University of Singapore · 2SAP
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6RL for Language Models Rubric-Based RL
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
Weight-Space Geometry of Offline Reasoning Training
Jun 21, 2026Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov +1RL for Language Models Direct Preference Optimization
1Keenable AI · 2Accenture · 3Yandex
Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
Jun 20, 2026Zhao Yang, Yuxuan Jiang, Ting-Chih Chen +18RL for Language Models Credit Assignment in RL
VU Amsterdam, The Netherlands · University of Maryland, Baltimore County, USA · Leiden University, The Netherlands +4

Jul 17, 2026, cs.LGJ/K move · Enter open · S save

QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

Zhengyang Zhuge, Hao Yu, Xin Wang, Zheng Li+3

Qwen Team, Alibaba Inc.

LLM Quantization RL for Language Models Mixture-of-Experts Language Models Quantization-Aware Training 4-Bit Quantization Activation Quantization

Open paper AskPDF arXiv

Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8. As an emerging low-precision format, NVFP4 combines fine-grained scaling for accuracy preservation with native W4A4 FP4 GEMMs for higher throughput than FP8. However, we find that directly applying NVFP4 to MoE RL rollout is impractical. NVFP4 rollout with BF16 training collapses after roughly 150 steps, accompanied by rapidly growing rollout-trainer log-probability gaps. Through training-inference error analysis and controlled ablations, we identify activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization-dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid. Therefore, to stabilize NVFP4 RL for MoE, we propose QUantization-error Alignment across Dual Sides (QUADS). On the trainer side, we introduce Asymmetric Quantization-Aware Training fake-quantizing weights while keeping activations unquantized for better alignment. On the rollout side, Residual Activation Compensation corrects high-error activation channels while preserving native W4A4 GEMMs. In our MoE RL experiments on several benchmarks, QUADS achieves BF16-level accuracy, improves average pass@1 by 21.49 points over naive NVFP4 RL, and delivers ~16% higher rollout throughput than FP8.

Momentum

Related topics

Latest papers 271