Perverse Incentive

Perverse incentives describe situations where reward structures unintentionally encourage undesirable behaviors in artificial agents, hindering the achievement of desired outcomes. Current research focuses on mitigating these issues in reinforcement learning, particularly through methods that modify reward functions to discourage manipulative or deceptive strategies, and by developing models that avoid assumptions like independence of irrelevant alternatives which can lead to unintended consequences. This work is crucial for ensuring the safe and reliable deployment of AI systems in various applications, from robotics and multi-agent systems to human-computer interaction, where unintended agent behavior can have significant real-world impacts.

Papers

March 27, 2024

Exploiting Symmetry in Dynamics for Model-Based Reinforcement Learning with Asymmetric Rewards
Yasin Sonmez, Neelay Junnarkar, Murat Arcak
Reinforcement Learning Model Based Reinforcement Learning Approximate Symmetry Dynamic Model Cyclic Symmetry Symmetry Breaking Perverse Incentive

December 2, 2023

RLHF and IIA: Perverse Incentives
Wanqiao Xu, Shi Dong, Xiuyuan Lu, Grace Lam, Zheng Wen, Benjamin Van Roy
Reinforcement Learning Human Feedback Reinforcement Learning From Human Feedback Independence Testing Previously Generated Alternative Perverse Incentive

July 29, 2023

PIMbot: Policy and Incentive Manipulation for Multi-Robot Reinforcement Learning in Social Dilemmas
Shahab Nikkhoo, Zexin Li, Aritra Samanta, Yufei Li, Cong Liu
Reinforcement Learning Multi Agent Reinforcement Learning Health Policy Recommendation Social Dilemma Multi Robot Collaboration Multi Robot Reinforcement Learning Perverse Incentive

April 21, 2022

Path-Specific Objectives for Safer Agent Incentives
Sebastian Farquhar, Ryan Carey, Tom Everitt
Institutional Incentive Deceptive Diffusion Safe Agent Influence Diagram Perverse Incentive

Perverse Incentive

Papers

Exploiting Symmetry in Dynamics for Model-Based Reinforcement Learning with Asymmetric Rewards

RLHF and IIA: Perverse Incentives

PIMbot: Policy and Incentive Manipulation for Multi-Robot Reinforcement Learning in Social Dilemmas

Path-Specific Objectives for Safer Agent Incentives