Adversarial Reward

Adversarial reward methods in machine learning aim to improve model performance and safety by training models to perform well even under challenging or manipulated reward signals. Current research focuses on developing robust algorithms and reward designs, often employing reinforcement learning frameworks and generative models, to mitigate vulnerabilities to adversarial attacks and improve the efficiency of training processes. This work is crucial for enhancing the reliability and safety of AI systems, particularly in applications like large language models and image generation, where robustness to unexpected inputs or malicious manipulations is paramount. The development of effective adversarial reward techniques is essential for building more trustworthy and dependable AI systems.

Papers

October 9, 2024

Self-Boosting Large Language Models with Synthetic Preference Data
Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, Furu Wei
Prompt Generation Synthetic Preference Preference Dataset Adversarial Reward

October 3, 2024

Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation
Rohin Manvi, Anikait Singh, Stefano Ermon
Large Language Model Inference Time Adversarial Reward Best of N

October 2, 2024

Generative Reward Models
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, Alon Albalak
Synthetic Preference Adversarial Reward Zero Shot LLM

September 19, 2024

Defending against Reverse Preference Attacks is Difficult
Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, Hassan Sajjad
Adversarial Reinforcement Learning Training Time Attack Adversarial Reward Preference Poisoning

August 31, 2024

AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation
Zanlin Ni, Yulin Wang, Renping Zhou, Rui Lu, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Yuan Yao, Gao Huang
Image Generation Reward Model Token Generation Adaptive Policy ImageNet E Non Autoregressive Transformer Adversarial Reward

May 23, 2024

Learning Constrained Markov Decision Processes With Non-stationary Rewards and Constraints
Francesco Emanuele Stradi, Anna Lunghi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti
Participation Constraint Bandit Feedback Constrained Markov Decision Process Stochastic Reward Adversarial Markov Decision Process Constraint Violation Adversarial Reward

February 21, 2024

Misalignment, Learning, and Ranking: Harnessing Users Limited Attention
Arpit Agarwal, Rad Niazadeh, Prathamesh Patil
LeArning Abstract Recommendation System Payoff Function Efficient Attention Instance Dependent Regret Segment Misalignment Adversarial Reward

January 11, 2024

Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, Ji-Rong Wen
Reinforcement Learning Fine Grained Adversarial Reward Editing Model Reinforcement Learning Objective

December 19, 2023

Resource-efficient Generative Mobile Edge Networks in 6G Era: Fundamentals, Framework and Case Study
Bingkun Lai, Jinbo Wen, Jiawen Kang, Hongyang Du, Jiangtian Nie, Changyan Yi, Dong In Kim, Shengli Xie
New Framework Case Study New Era Edge Intelligence Basic Concept Resource Constrained Edge Adversarial Reward

June 2, 2023

PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward
Weichao Zhou, Wenchao Li
Imitation Learning Inverse Reinforcement Learning Adversarial Reward Hybrid Reward Semi Supervised Reward

May 22, 2023

Measuring and Modeling Physical Intrinsic Motivation
Julio Martinez, Felix Binder, Haoliang Wang, Nick Haber, Judith Fan, Daniel L. K. Yamins
Measurement System Intrinsic Motivation Adversarial Reward Physic Prediction

May 9, 2023

Fine-tuning Language Models with Generative Adversarial Reward Modelling
Zhang Ze Yu, Lau Jia Jaw, Zhang Hui, Bryan Kian Hsiang Low
Reinforcement Learning Adversarial Training Supervised Fine Tuning AI Alignment Language Model Fine Tuning Adversarial Reward Adversarial Feedback

February 14, 2023

Adversarial Rewards in Universal Learning for Contextual Bandits
Moise Blanchard, Steve Hanneke, Patrick Jaillet
Multi Armed Bandit Contextual Bandit Sublinear Regret Universal Learning Adversarial Reward

September 29, 2022

GROOT: Corrective Reward Optimization for Generative Sequential Labeling
Kazuma Hashimoto, Karthik Raman
NLP Task Sequence Tagging Adversarial Reward

June 28, 2022

GAN-based Intrinsic Exploration For Sample Efficient Reinforcement Learning
Doğay Kamar, Nazım Kemal Üre, Gözde Ünal
Reinforcement Learning Sparse Reward Efficient Exploration Sample Efficient Reinforcement Learning Adversarial Reward Intrinsic Exploration

June 27, 2022

On the Complexity of Adversarial Decision Making
Dylan J. Foster, Alexander Rakhlin, Ayush Sekhari, Karthik Sridharan
Adversarial Attack Complexity Matter Adversarial Markov Decision Process Adversarial Reward General Adversarial

March 23, 2022

Your Policy Regularizer is Secretly an Adversary
Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, Pedro Ortega
Reinforcement Learning Adversary Agent Policy Regularization Adversarial Reward