RL Fine-Tuning

RL: Reinforcement Learning

Reinforcement learning fine-tuning (RFT) adapts pretrained language, vision-language, and control models by updating them from feedback on generated answers or actions rather than relying only on demonstrations. Common approaches include PPO and GRPO for outcome-based learning, RL with verifiable rewards for math and code, and preference or rubric-based methods for open-ended tasks; curricula, reward models, and step-level credit assignment help make feedback more useful. Work also extends to tool-using agents, robots, diffusion policies, and offline-to-online control, where methods such as value learning, residual policies, and flow-matching optimization address costly or sparse environment feedback. Key challenges are designing reliable rewards, balancing exploration with stable updates, reducing rollout costs, and ensuring gains transfer across tasks without forgetting prior capabilities or exploiting the evaluator.

Latest papers

233 papers17 papers in the last four weeks, down 23% on the four weeks before. 0.1% of all new papers.

Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
Apr 12, 2026Subramanyam SahooLanguage Model Calibration Reward Hacking
Cambridge AI Safety Hub (CAISH)
E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning
Apr 10, 2026Weiyang Guo, Zesheng Shi, Liye Zhao +5Tool-Integrated Reasoning LLM Agent Training
Harbin Institute of Technology, Shenzhen, China · Huawei Technologies Co., Ltd. · Hong Kong University of Science and Technology
Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
Apr 10, 2026Taojie Zhu, Dongyang Xu, Ding Zou +4LLM Fine-Tuning RL for Language Model Reasoning
Shenzhen International Graduate School, Tsinghua University · Intelligent System Department, Zhongxing Telecom Equipment (ZTE) · Institute of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications
QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
Apr 9, 2026Hao Gu, Hao Wang, Jiacheng Liu +9Reinforcement Learning RL Fine-Tuning
The Hong Kong University of Science and Technology · City University of Hong Kong · Zhejiang University
ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuning
Apr 9, 2026Jiani Huang, Shijie Wang, Liangbo Ning +2Large Language Model-Based Recommendation RL Fine-Tuning
The Hong Kong Polytechnic University
ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training
Apr 8, 2026Yu Liang, Liangxin Liu, Longzheng Wang +5Reward Modeling LLM Alignment
Baidu Inc., Beijing, China
Discrete Flow Matching Policy Optimization
Apr 7, 2026Maojiang Su, Po-Chung Hsieh, Weimin Wu +4TV Regularization RL for Flow Matching Models
Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Electrical Engineering, National Taiwan University, Taipei 10617, Taiwan · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1
Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning
Apr 7, 2026Navan Preet Singh, Xiaokun Wang, Anurag Garikipati +3LLM Fine-Tuning Language Modeling
Forta, Houston, TX · East China Normal University, Shanghai, China · Incept Labs, Houston, TX +1
From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
Apr 1, 2026Rui Wu, Ruixiang TangReward Hacking RL Fine-Tuning
Department of Computer Science Rutgers University
RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning
Apr 1, 2026Shaopeng Fu, Xingxing Zhang, Li Dong +1RL for Code Generation LLM Self-Refinement
Work done during an internship at Microsoft Research · KAUST · Microsoft Research
MotionRFT: Unified Reinforcement Fine-Tuning for Text-to-Motion Generation
Mar 28, 2026Xiaofeng Tan, Wanjiang Weng, Hongsong Wang +3Text-to-Motion Generation Human Motion Generation
School of Computer Science and Engineering, Southeast University, Nanjing 211189, China, and also with Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · State Key Laboratory for Novel Software Technology, School of Intelligence Science and Technology, Nanjing University, Nanjing 210023, China · New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), and also with School of Artificial Intelligence, University of Chinese Academy of Sciences
PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learning
Mar 25, 2026Yankai Wang, Yiding Sun, Qirui Wang +3Point Cloud Learning RL Fine-Tuning
School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · International School, Beijing University of Posts and Telecommunications, Beijing 100876, China
Off-Policy Value-Based Reinforcement Learning for Large Language Models
Mar 24, 2026Peng-Yuan Wang, Ziniu Li, Tian Xu +8Reinforcement Learning RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data +1
Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
Mar 23, 2026Haoming Meng, Kexin Huang, Shaohang Wei +6RL for Language Model Reasoning RL Fine-Tuning
Alibaba Group
Fine-tuning Timeseries Predictors Using Reinforcement Learning
Mar 20, 2026Hugo Cazaux, Ralph Rudd, Hlynur Stefánsson +2RL Fine-Tuning Time Series Forecasting
Reykjavik University, Department of Engineering, Menntavegur 1, Reykjavik, 102, Iceland
Learning to Reason with Curriculum I: Provable Benefits of Autocurriculum
Mar 18, 2026Nived Rajaraman, Audrey Huang, Miro Dudik +3Efficient Language Model Training Curriculum RL
Microsoft Research · University of Illinois Urbana Champaign
Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
Mar 18, 2026Hao Ma, Zhiqiang Pu, Yang Liu +1RL Fine-Tuning RL for Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences
SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion
Mar 17, 2026Elham Daneshmand, Shafeef Omar, Glen Berseth +2RL Fine-Tuning RL for Legged Locomotion
McGill University · Mila - Quebec AI Institute · Technical University of Munich +1
Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
Mar 17, 2026Yongyu Mu, Jiali Zeng, Fandong Meng +2LLM Mathematical Reasoning Supervised Fine-Tuning
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China
MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
Mar 13, 2026Tianyang Luo, Tao Feng, Zhigang Hua +4RL for Language Model Reasoning RL Fine-Tuning
University of Illinois Urbana-Champaign · Meta
Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies
Mar 13, 2026Junhe Sheng, Ruofei Bai, Kuan Xu +5Robot Navigation Robot Policy Adaptation
Nanyang Technological University, Singapore. · A*STAR, Singapore. · National University of Singapore, Singapore.
Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agents
Mar 13, 2026Jiawei Huang, Qingping Yang, Renjie Zheng +1Software Engineering Agents Reward Design for RL
ETH Zurich · ByteDance Seed
Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
Mar 12, 2026Samy Jelassi, Mujin Kwun, Rosie Zhao +5Fine-Tuning Language Models LLM Fine-Tuning
Harvard University, Kempner Institute · MBZUAI · Microsoft Research New England
Can RL Improve Generalization of LLM Agents? An Empirical Study
Mar 12, 2026Zhiheng Xi, Xin Guo, Jiaqi Liu +11LLM Agents RL for LLM Agents
Fudan University · Shanghai Artificial Intelligence Laboratory · Meituan
DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning
Mar 11, 2026Hanxu Hu, Yuxuan Wang, Maggie Huan +4RL for Language Model Reasoning RL Fine-Tuning
University of Zurich · University of Pennsylvania · ETH Zurich +2
Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
Mar 11, 2026Yixiu Mao, Yun Qu, Qi Wang +2Prompt Selection for RL RL for Language Model Reasoning
Department of Automation, Tsinghua University
Fibration Policy Optimization
Mar 9, 2026Chang Li, Tshihao Tsu, Yaren Zhang +2Policy Optimization RL for Language Models
JD Explore Academy · Carleton University
NuNext: Reframing Nucleus Detection as Next-Point Detection
Mar 7, 2026Zhongyi Shui, Honglin Li, Xiaozhong Ji +7Computational Pathology Object Detection
Zhejiang University · Westlake University · Shanghai Artificial Intelligence Laboratory +3
Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspaces
Mar 5, 2026Karan Gupta, Pranav Vajreshwari, Yash Pandya +3RL for Tool Use Agentic RL
Microsoft Research
CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning
Mar 3, 2026Zhenquan Yao, Zitong Huang, Yihan Zeng +5Continual Learning for LLM Agents RL for GUI Agents
Harbin Institute of Technology · Huawei Noah’s Ark Lab · University College Dublin +1

Apr 12, 2026J/K move · Enter open · S save

Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs

Subramanyam Sahoo

Cambridge AI Safety Hub (CAISH)

Language Model Calibration Reward Hacking RL Fine-Tuning LLM Sycophancy

Open paper AskPDF arXiv

Modern large language models (LLMs) are increasingly fine-tuned via reinforcement learning from human feedback (RLHF) or related reward optimisation schemes. While such procedures improve perceived helpfulness, we investigate whether sycophantic reward signals degrade calibration -- a property essential for reliable uncertainty quantification. We fine-tune Qwen3-8B under three regimes: no fine-tuning (base), neutral supervised fine-tuning (SFT) on TriviaQA, and sycophancy-inducing Group Relative Policy Optimisation (GRPO) that rewards agreement with planted wrong answers. Evaluating on

1{,}000

MMLU items across five subject domains with bootstrap confidence intervals and permutation testing, we find that sycophantic GRPO produces consistent directional calibration degradation -- ECE rises by

+0.006

relative to the base model and MCE increases by

+0.010

relative to neutral SFT -- though the effect does not reach statistical significance (

p = 0.41

) at this training budget. Post-hoc matrix scaling applied to all three models reduces ECE by

40

64\%

and improves accuracy by

1.5

3.0

percentage points. However, the sycophantic model retains the highest post-scaling ECE relative to the neutral SFT control (

0.042

vs.\

0.037

), suggesting that reward-induced miscalibration leaves a structured residual even after affine correction. These findings establish a methodology for evaluating the calibration impact of reward hacking and motivate calibration-aware training objectives.