Value Function Factorization

Value function factorization addresses the challenge of scaling multi-agent reinforcement learning (MARL) by decomposing a global value function into individual agent utilities, enabling decentralized execution while maintaining centralized training. Current research focuses on improving the expressiveness and efficiency of factorization methods, exploring architectures like QMIX and its variants (e.g., incorporating maximum entropy, weighted losses, or non-monotonic functions), as well as leveraging graph neural networks and counterfactual predictions to enhance performance. These advancements are significant for improving the scalability and applicability of MARL to complex cooperative tasks, particularly in domains like robotics and game playing.

Papers

August 27, 2024

On Stateful Value Factorization in Multi-Agent Reinforcement Learning
Enrico Marchesini, Andrea Baisero, Rupali Bhati, Christopher Amato
Multi Agent Reinforcement Learning Utility Maximization StarCraft II Value Function Factorization

June 20, 2024

Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization
Wentse Chen, Shiyu Huang, Jeff Schneider
Multi Agent Reinforcement Learning Maximum Entropy Deterministic Policy Maximum Entropy Reinforcement Learning Value Function Factorization Soft QMIX

May 13, 2024

POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
Chang Huang, Junqiao Zhao, Shatong Zhu, Hongtu Zhou, Chen Ye, Tiantian Feng, Changjun Jiang
Multi Agent Cooperative Multi Agent Reinforcement Learning Action Recognition Model Value Function Factorization Weight Factorization

March 19, 2024

Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation
Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, Guoqing Zhao
Large Language Model Sign Language Translation Gloss Translation Value Function Factorization

December 24, 2023

ConcaveQ: Non-Monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning
Huiqun Li, Hanhan Zhou, Yifei Zou, Dongxiao Yu, Tian Lan
Multi Agent Reinforcement Learning Convex Set Action Value Function ConvQA Face Challenge Value Function Factorization

November 1, 2023

QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning
Rizhong Wang, Huiping Li, Di Cui, Demin Xu
Multi Agent Reinforcement Learning Multi Agent Challenge Matrix Game Value Function Factorization

October 19, 2023

Impact of Relational Networks in Multi-Agent Learning: A Value-Based Factorization View
Yasin Findik, Paul Robinette, Kshitij Jerath, S. Reza Ahmadzadeh
Global Impact Multi Agent Learning Inter Agent Agent Learning Value Function Factorization Relational Network Real World Multi Agent

June 4, 2023

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning
Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee
Multi Agent Reinforcement Learning Unified Framework Distributional Reinforcement Learning Multi Agent Challenge Stochastic Reward Value Function Factorization

February 11, 2023

ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning
Yongsheng Mei, Hanhan Zhou, Tian Lan
Regret Minimization Action Value Function Value Function Factorization

June 22, 2022

PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning
Hanhan Zhou, Tian Lan, Vaneet Aggarwal
Multi Agent Reinforcement Learning Counterfactual Prediction Utility Maximization Action Value Function Value Function Factorization

June 2, 2022

RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning
Hao Chen, Guangkai Yang, Junge Zhang, Qiyue Yin, Kaiqi Huang
Reinforcement Learning Multi Agent Deep Reinforcement Learning Ad Hoc Value Function Factorization

May 30, 2022

Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning
Rafael Pina, Varuna De Silva, Joosep Hook, Ahmet Kondoz
Reinforcement Learning Multi Agent Reinforcement Learning Multi Agent Multi Agent Setting Residual Reinforcement Learning Action Value Function Value Function Factorization

May 25, 2022

QGNN: Value Function Factorisation with Graph Neural Networks
Ryan Kortvelesy, Amanda Prorok
Graph Neural Network Multi Agent Reinforcement Learning Value Function Value Function Factorization Quantum Graph Neural Network

January 4, 2022

Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
Hanhan Zhou, Tian Lan, Vaneet Aggarwal
Value Function Latent State Soft Actor Critic Parameter Sharing Value Function Factorization Decentralized Policy Gradient

Value Function Factorization

Papers

On Stateful Value Factorization in Multi-Agent Reinforcement Learning

Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization

POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning

Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation

ConcaveQ: Non-Monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning

QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning

Impact of Relational Networks in Multi-Agent Learning: A Value-Based Factorization View

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning

ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning

PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning

RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning

Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning

QGNN: Value Function Factorisation with Graph Neural Networks

Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients