RL from Human Feedback

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 84

All topics

Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
Jun 9, 2026Guozheng Li, Xiyan Fu, Yiwen GuoRL for Language Models Sample-Efficient RL
Southeast University · Nanyang Technological University · Independent Researcher
A Regret Minimization Framework on Preference Learning in Large Language Models
Jun 8, 2026Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim +4Regret Minimization in RL LLM Alignment
Seoul National University, Seoul, South Korea · LG AI Research, Seoul, South Korea · UNIST, Ulsan, South Korea +1
A Unifying Lens on Reward Uncertainty in RLHF
Jun 8, 2026Ely Hahami, Yoel Zimmermann, Ray Zhou +1Reward Modeling KL-Regularized RL
Harvard College, Harvard University, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance
Jun 8, 2026Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova +1LLM Alignment Text Summarization
Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC), Mexico City, Mexico
Variational Proximal Policy Optimization
Jun 6, 2026Ousmane Amadou DiaRL for Language Model Reasoning Proximal Policy Optimization
EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms
Jun 2, 2026Guilin Zhang, Chuanyi Sun, Kai Zhao +3Early Stopping Reinforcement Learning
Workday AI Research · The George Washington University, Washington, DC, USA.
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
Jun 2, 2026Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan +1Reward Hacking LLM Evaluation
College of Engineering, The Pennsylvania State University, University Park, PA, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA · 3Penn State Great Valley, The Pennsylvania State University, Malvern, PA, USA +1
Efficient Exploration for Iterative Nash Preference Optimization
May 31, 2026Tianlong Nan, Xiaopeng Li, Christian Kroer +1Nash Equilibrium LLM Alignment
Columbia University⋄ · The Chinese University of Hong Kong, Shenzhen†
The Representation-Rationalizability Tradeoff in Reward Learning
May 29, 2026Jing Dong, Yaoliang Yu, Pascal PourpartPairwise Preference Learning Reward Modeling
Vector Institute · University of Waterloo & Vector Institute
In-Context Reward Adaptation for Robust Preference Modeling
May 28, 2026Zhenyu Sun, Zheng Xu, Ermin WeiHuman Preference Modeling In-Context Learning
Northwestern University · Meta Superintelligence Labs
Training Stratigraphy: Persistent Behavioral Artifacts in Large Language Models Observed Through Longitudinal AI-Human Interaction
May 27, 2026Chen Ying Claude, Zhihan LuoLLM Auditing Human-in-the-Loop AI
*Anthropic · †Independent Researcher
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
May 26, 2026Dongyoon Hahm, Dylan Hadfield-Menell, Kimin LeeLLM Alignment Data Poisoning Attacks
1KAIST · 2MIT
Learning Kernel-Based MDPs from Episodic Preferential Feedback
May 22, 2026Nikola Pavlovic, Sattar Vakili, Qing ZhaoPairwise Preference Learning Regret Minimization in RL
Cornell University · MediaTek Research
Echo: Learning from Experience Data via User-Driven Refinement
May 21, 2026Hande Dong, Xiaoyun Liang, Jiarui Yu +15Continual Learning for LLM Agents LLM Agent Self-Improvement
Tencent, CodeBuddy
Implicit Safety Alignment from Crowd Preferences
May 20, 2026Qian Lin, Daniel S. BrownSafe RL Preference-Based RL
Kahlert School of Computing, University of Utah
Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM Alignment Direct Preference Optimization
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
Reinforcing Human Behavior Simulation via Verbal Feedback
May 19, 2026Weiwei Sun, Xuhui Zhou, Jiarui Liu +13RL for Language Models Human Behavior Simulation
1Carnegie Mellon University · 2Microsoft
ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks
May 17, 2026Tianxiang Xu, Xiaoyan Zhu, Xin Lai +1Reward Modeling Reinforcement Learning
School of Computer Science and Technology, Xi’an Jiaotong University
Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedback
May 15, 2026Pengcheng Dai, He Wang, Dongming Wang +2Policy Gradient Decentralized MARL
Singapore University of Technology and Design, Singapore 487372, Singapore · Southeast University, Nanjing 211189, China · University of California, Riverside, CA 92521, USA +2
GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zero
May 14, 2026Shangjian Yin, Yu Fu, Yue Dong +1Reinforcement Learning RL for Language Model Reasoning
University of California, Riverside
Semantic Reward Collapse and the Preservation of Epistemic Integrity in Adaptive AI Systems
May 12, 2026William ParrisReward Modeling Reward Hacking
BDB Labs / BagelTech · San Diego, California, USA
When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
May 12, 2026Xiaofeng Tan, Jun Liu, Bin-Bin Gao +5Flow Matching Generative Modeling
Southeast University · Tencent Youtu Lab · Southern University of Science and Technology
$f$ -Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
May 7, 2026Di Wu, Chengshuai Shi, Jing Yang +1RL for Language Models RL from Human Feedback
Department of Electrical and Computer Engineering, University of Virginia, Virginia, United States · Princeton Language and Intelligence, Princeton University, New Jersey, United States
Mitigating Cognitive Bias in RLHF by Altering Rationality
May 7, 2026Tiffany Horter, Andrew Markham, Niki Trigoni +1Human Preference Modeling RL from Human Feedback
University of Oxford · Brown University
Optimal Transport for LLM Reward Modeling from Noisy Preference
May 7, 2026Licheng Pan, Haochen Yang, Haoxuan Li +8Reward Modeling Noisy-Label Learning
Zhejiang University · Xiaohongshu Inc · National University of Singapore +2
Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
May 6, 2026Zhen-Yu Zhang, Yuting Tang, Jiandong Zhang +2RL Exploration Sample-Efficient RL
Center for Advanced Intelligence Project, RIKEN · Graduate School of Frontier Sciences, The University of Tokyo · Northeastern University +1
Explaining and Preventing Alignment Collapse in Iterative RLHF
May 5, 2026Etienne Gauthier, Francis Bach, Michael I. JordanReward Hacking Policy Gradient
Inria, Ecole Normale Sup´erieure, PSL Research University · Ecole Normale Sup´erieure, PSL Research University · PSL Research University +1
Efficient Preference Poisoning Attack on Offline RLHF
May 4, 2026Chenye Yang, Weiyu Xu, Lifeng LaiDirect Preference Optimization Data Poisoning Attacks
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, USA.
PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs
May 1, 2026Ravi Ranjan, Utkarsh Grover, Xiaomin Lin +1Automated Code Review RL from Human Feedback
Florida International University (FIU), Miami, USA
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
Apr 30, 2026Yikai Wang, Shang Liu, Jose BlanchetRegret Minimization in RL Reinforcement Learning
Department of Statistics and Operations Research, University of North Carolina · Imperial Business School, Imperial College London · Department of Management Science and Engineering, Stanford University

Jun 9, 2026, cs.LGJ/K move · Enter open · S save

Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output

Guozheng Li, Xiyan Fu, Yiwen Guo

Southeast University · Nanyang Technological University · Independent Researcher

RL for Language Models Sample-Efficient RL RL from Human Feedback

Open paper AskPDF arXiv

Current reinforcement learning from human feedback (RLHF) methods primarily rely on scalar rewards from a trained reward model (RM). While effective, scalar rewards are often noisy and fail to capture fine-grained preference differences, whereas RM hidden states encode richer semantic and preference information. We introduce the representation-aware advantage estimation, which leverages RM hidden states and models them as auxiliary signals for better advantage estimation. Specifically, we propose the Graph-based Advantage Estimation (GraphAE), treat each sampled group as a graph, where nodes correspond to responses and edges capture their similarity in the RM hidden space. Then advantages are computed via graph propagation, enabling each sample to incorporate contextual information from its neighbors. GraphAE is lightweight and can be seamlessly integrated into existing group-based RL algorithms. We apply GraphAE to GRPO, GSPO and RLOO, and conduct extensive experiments on different models and benchmarks. Empirical results show consistent improvements across three benchmarks, with gains of up to + 6.3 on Arena-Hard-v0.1, + 8.27 on AlpacaEval 2.0, and + 0.22 on MT-Bench. These results demonstrate that leveraging RM representations leads to more sample efficient and robust RLHF.

Momentum

Related topics

Latest papers 84