Policy Alignment

Policy alignment in artificial intelligence focuses on ensuring that AI systems' actions and goals align with human values and preferences. Current research emphasizes developing efficient algorithms, such as those based on reinforcement learning with human feedback (RLHF) and constrained optimization, to learn policies that satisfy both reward maximization and adherence to human-specified constraints or preferences. These efforts leverage various model architectures, including large language models and spiking neural networks, and address challenges like data efficiency, distribution shifts, and the interpretability of learned policies. The ultimate goal is to create trustworthy and beneficial AI systems by bridging the gap between AI objectives and human intentions, with implications for safety, fairness, and the responsible deployment of AI in diverse applications.

Papers

November 16, 2024

Adaptive Learning of Design Strategies over Non-Hierarchical Multi-Fidelity Models via Policy Alignment
Akash Agrawal (1), Christopher McComb (1) ((1) Carnegie Mellon University)
Adaptive Learning Multi Fidelity Design Principle Fidelity Model Policy Alignment

October 17, 2024

Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, Sergey Levine
Non Humanoid Robot Net Present Value Human Guidance Robot Policy Policy Alignment Generalist Learner Different Robot

June 11, 2024

Joint Demonstration and Preference Learning Improves Policy Alignment with Human Feedback
Chenliang Li, Siliang Zeng, Zeyi Liao, Jiaxiang Li, Dongyeop Kang, Alfredo Garcia, Mingyi Hong
Reinforcement Learning Human Feedback Reward Model Preference Learning Textual Demonstration Policy Alignment Direct Policy

June 9, 2024

Information Theoretic Guarantees For Policy Alignment In Large Language Models
Youssef Mroueh
Large Language Model Constrained Policy Optimization Optimal Alignment Policy Alignment R\'enyi Divergence

June 7, 2024

Algorithms for learning value-aligned policies considering admissibility relaxation
Andrés Holgado-Sánchez, Joaquín Arias, Holger Billhardt, Sascha Ossowski
Practical Algorithm Participation Constraint Value Alignment Policy Alignment

May 30, 2024

TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models
Chen Zhang, Chengguang Tang, Dading Chong, Ke Shi, Guohua Tang, Feng Jiang, Haizhou Li
Large Language Model Instruction Following LD Align Teacher Model Teacher Student Policy Alignment Policy Fine Tuning

May 29, 2024

Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning
Hanye Zhao, Xiaoshen Han, Zhengbang Zhu, Minghuan Liu, Yong Yu, Weinan Zhang
Diffusion Model Offline Reinforcement Learning Dynamic Model Policy Alignment Model Rollouts Diffusion Dynamic

May 28, 2024

AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization
Longxiang He, Li Shen, Junbo Tan, Xueqian Wang
Constrained Optimization Implicit Language Q Function Policy Alignment Implicit Q Learning Implicit Policy

January 25, 2024

Learning fast changing slow in spiking neural networks
Cristiano Capone, Paolo Muratore
Reinforcement Learning Neural Network LeArning Abstract Proximal Policy Optimization Experience Replay Stability Plasticity Policy Alignment

January 18, 2024

The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learning
Ali Baheri, Mykel J. Kochenderfer
Multi Agent Reinforcement Learning Optimal Transport Multi Agent Learning Model Synergy Policy Alignment Agent Strategy

August 3, 2023

PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback
Souradip Chakraborty, Amrit Singh Bedi, Alec Koppel, Dinesh Manocha, Huazheng Wang, Mengdi Wang, Furong Huang
Reinforcement Learning Unified Framework Human Feedback Bilevel Optimization Stochastic Bilevel Optimization Policy Alignment

June 9, 2023

Combining a Meta-Policy and Monte-Carlo Planning for Scalable Type-Based Reasoning in Partially Observable Environments
Jonathon Schwartz, Hanna Kurniawati, Marcus Hutter
Multi Agent System Autonomous Agent Monte Carlo Tree Search Observable Environment Policy Alignment Observable Non Deterministic Planning Monte Carlo Planning

May 27, 2023

Query-Policy Misalignment in Preference-Based Reinforcement Learning
Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, Ya-Qin Zhang
Reinforcement Learning Policy Learning Preference Based Reinforcement Learning Policy Alignment

May 22, 2023

An ASP Framework for the Refinement of Authorization and Obligation Policies
Daniela Inclezan
Answer Set Programming Prior Policy Policy Design Policy Alignment

January 30, 2023

January 27, 2023

Policy-Value Alignment and Robustness in Search-based Multi-Agent Learning
Niko A. Grupen, Michael Hanlon, Alexis Hao, Daniel D. Lee, Bart Selman
Native Robustness Multi Agent Learning Policy Alignment

December 22, 2022

Local Policy Improvement for Recommender Systems
Dawen Liang, Nikos Vlassis
Recommender System Policy OpTimization Sequential Recommendation Policy Alignment

October 6, 2022

Distributionally Adaptive Meta Reinforcement Learning
Anurag Ajay, Abhishek Gupta, Dibya Ghosh, Sergey Levine, Pulkit Agrawal
Meta Reinforcement Learning Distributional Robustness Policy Alignment Meta Reinforcement Meta Rl Algorithm

September 21, 2022

On the Convergence Theory of Meta Reinforcement Learning with Personalized Policies
Haozhi Wang, Qing Wang, Yunfeng Shao, Dong Li, Jianye Hao, Yinchuan Li
Policy Gradient Meta Reinforcement Learning Policy Alignment Meta Rl Algorithm Convergence Theory Online Policy Customization

Policy Alignment

Papers

Adaptive Learning of Design Strategies over Non-Hierarchical Multi-Fidelity Models via Policy Alignment

Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance

Joint Demonstration and Preference Learning Improves Policy Alignment with Human Feedback

Information Theoretic Guarantees For Policy Alignment In Large Language Models

Algorithms for learning value-aligned policies considering admissibility relaxation

TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models

Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning

AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization

Learning fast changing slow in spiking neural networks

The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learning

PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback

Combining a Meta-Policy and Monte-Carlo Planning for Scalable Type-Based Reasoning in Partially Observable Environments

Query-Policy Misalignment in Preference-Based Reinforcement Learning

An ASP Framework for the Refinement of Authorization and Obligation Policies

Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs

Transferring Multiple Policies to Hotstart Reinforcement Learning in an Air Compressor Management Problem

Policy-Value Alignment and Robustness in Search-based Multi-Agent Learning

Local Policy Improvement for Recommender Systems

Distributionally Adaptive Meta Reinforcement Learning

On the Convergence Theory of Meta Reinforcement Learning with Personalized Policies