Preference Poisoning

Preference poisoning attacks exploit vulnerabilities in machine learning systems, particularly those trained using human feedback, by manipulating preference data to induce undesirable behaviors. Current research focuses on understanding the effectiveness of various poisoning strategies across different model architectures, including those based on reinforcement learning and federated learning, and exploring defensive mechanisms such as constrained optimization and robust loss functions. This research is crucial for ensuring the safety and reliability of AI systems, especially those impacting decision-making in high-stakes applications like recommendation systems and search engines, where malicious actors could manipulate user preferences for their own gain.

Papers

September 19, 2024

Defending against Reverse Preference Attacks is Difficult
Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, Hassan Sajjad
Adversarial Reinforcement Learning Training Time Attack Adversarial Reward Preference Poisoning

August 1, 2024

Exploiting Preferences in Loss Functions for Sequential Recommendation via Weak Transitivity
Hyunsoo Chung, Jungtaek Kim, Hyungeun Jo, Hyungwon Choi
Loss Function Recommender System Sequential Recommendation Movie Recommendation Strong Transitivity Relation Optimization Objective Binary Label Preference Poisoning

June 26, 2024

Adversarial Search Engine Optimization for Large Language Models
Fredrik Nestaas, Edoardo Debenedetti, Florian Tramèr
Adversarial Search Preference Poisoning

April 8, 2024

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler
Reinforcement Learning From Human Feedback Reward Model Training Best of N Poisoned Data Preference Poisoning

February 2, 2024

Preference Poisoning Attacks on Reward Model Learning
Junlin Wu, Jiongxiao Wang, Chaowei Xiao, Chenguang Wang, Ning Zhang, Yevgeniy Vorobeychik
Reward Model Preference Learning Pairwise Comparison Preference Rating Attack Algorithm Preference Poisoning

November 16, 2023

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao
Reinforcement Learning Human Feedback Adversarial Learning Poisoning Attack RLHF V Reward Poisoning Attack Preference Poisoning

March 20, 2022

Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI
Matija Franklin, Hal Ashton, Rebecca Gorman, Stuart Armstrong
Artificial Intelligence Importance Aware Preference Feedback Speech Based Age Preference Change Preference Poisoning Preference Exploration

February 10, 2022

PPA: Preference Profiling Attack Against Federated Learning
Chunyi Zhou, Yansong Gao, Anmin Fu, Kai Chen, Zhiyang Dai, Zhi Zhang, Minhui Xue, Yuqing Zhang
PRIvacy Leakage Inference Attack Privacy Inference Preference Poisoning