Inverse Q Learning

Inverse Q-learning aims to infer reward functions from observed expert behavior, essentially reversing the standard reinforcement learning process. Current research focuses on developing robust algorithms, such as hierarchical inverse Q-learning (HIQL) and least squares inverse Q-learning (LS-IQ), that address challenges like handling absorbing states and learning from diverse, potentially unlabeled data. These advancements are improving the interpretability of learned behaviors, particularly in applications like understanding animal decision-making and enabling more effective offline reinforcement learning for complex tasks, such as goal-conditioned robotics. The resulting insights are valuable for both scientific understanding of decision-making processes and the development of more efficient and robust AI systems.

Papers

October 10, 2024

UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations
Huy Hoang, Tien Mai, Pradeep Varakantham
Inverse Reinforcement Learning Noisy Demonstration Expert Demonstration Inverse Q Learning

April 18, 2024

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function
Rafael Rafailov, Joey Hejna, Ryan Park, Chelsea Finn
Language Model Reinforcement Learning Direct Preference Optimization Q Function Implicit Reward Inverse Q Learning

November 23, 2023

Multi-intention Inverse Q-learning for Interpretable Behavior Representation
Hao Zhu, Brice De La Crompe, Gabriel Kalweit, Artur Schneider, Maria Kalweit, Ilka Diester, Joschka Boedecker
Inverse Reinforcement Learning Human Decision Behavior Representation Interpretable Reward Inverse Q Learning

July 22, 2023

HIQL: Offline Goal-Conditioned RL with Latent States as Actions
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, Sergey Levine
Reinforcement Learning Value Function Latent State Past Action Goal Conditioned Reinforcement Learning Unsupervised Pre Training Offline Goal Conditioned Reinforcement Learning Standardized Benchmark Inverse Q Learning

March 1, 2023

LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning
Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, Jan Peters
Reward Function Inverse Reinforcement Learning Implicit Reward Inverse Q Learning

Inverse Q Learning

Papers

UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Multi-intention Inverse Q-learning for Interpretable Behavior Representation

HIQL: Offline Goal-Conditioned RL with Latent States as Actions

LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning