Action Value Function

Action-value functions, which estimate the expected cumulative reward for taking a specific action in a given state, are central to reinforcement learning (RL). Current research focuses on improving the efficiency and accuracy of estimating these functions, particularly through advancements in model architectures like Q-networks and their variations (e.g., iterated Q-networks, residual Q-networks), value decomposition methods for multi-agent systems, and the incorporation of world models. These improvements are crucial for scaling RL to complex, high-dimensional problems and enabling its application in diverse fields such as robotics, healthcare, and marketing, where interpretability and sample efficiency are paramount.

Papers

June 28, 2024

Operator World Models for Reinforcement Learning
Pietro Novelli, Marco Pratticò, Massimiliano Pontil, Carlo Ciliberto
Reinforcement Learning Sequential Decision Policy Mirror Descent Action Value Function

April 3, 2024

Reinforcement Learning in Categorical Cybernetics
Jules Hedges, Riu Rodríguez Sakamoto
Reinforcement Learning Value Iteration Category Theory Action Value Function Bellman Update

April 1, 2024

Data-Driven Knowledge Transfer in Batch $Q^*$ Learning
Elynn Chen, Xi Chen, Wenbo Jing
Markov Decision Process Knowledge Transfer Data Driven Decision Action Value Function Dynamic Decision Batch Greenkhorn Q$ Learning Algorithm

March 4, 2024

Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
Théo Vincent, Daniel Palenicek, Boris Belousov, Jan Peters, Carlo D'Eramo
Reinforcement Learning Deep Reinforcement Learning Actor Critic Q Network Action Value Function Bellman Operator Bellman Update

February 5, 2024

Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent
Yingru Li, Jiawei Xu, Lei Han, Zhi-Quan Luo
Reinforcement Learning Practice Mode Bridging Text Exploration Problem Action Value Function

December 24, 2023

ConcaveQ: Non-Monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning
Huiqun Li, Hanhan Zhou, Yifei Zou, Dongxiao Yu, Tian Lan
Multi Agent Reinforcement Learning Convex Set Action Value Function ConvQA Face Challenge Value Function Factorization

November 22, 2023

Probabilistic Inference in Reinforcement Learning Done Right
Jean Tarbouriech, Tor Lattimore, Brendan O'Donoghue
Reinforcement Learning Markov Decision Process Action Value Function Entropy Exploration

September 8, 2023

Leveraging World Model Disentanglement in Value-Based Multi-Agent Reinforcement Learning
Zhizun Wang, David Meger
Multi Agent Multi Agent System Disentanglement Capability Value Decomposition Action Value Function Model Based Multi Agent Reinforcement

August 25, 2023

Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery
Patrick Emedom-Nnamdi, Timothy R. Smith, Jukka-Pekka Onnela, Junwei Lu
Reinforcement Learning Application Proficiency Prompt Recovery Action Value Function Interpretable Function

June 24, 2023

TVDO: Tchebycheff Value-Decomposition Optimization for Multi-Agent Reinforcement Learning
Xiaoliang Hu, Pengcheng Guo, Chuanwei Zhou, Tong Zhang, Zhen Cui
Multi Agent Reinforcement Learning Multi Objective Value Decomposition Action Value Function Value Function Decomposition

March 9, 2023

An Improved Data Augmentation Scheme for Model Predictive Control Policy Approximation
Dinesh Krishnamoorthy
Data Augmentation Effective Data Augmentation Action Value Function

February 11, 2023

ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning
Yongsheng Mei, Hanhan Zhou, Tian Lan
Regret Minimization Action Value Function Value Function Factorization

November 14, 2022

Linear Reinforcement Learning with Ball Structure Action Space
Zeyu Jia, Randy Jia, Dhruv Madeka, Dean P. Foster
Reinforcement Learning Optimal Policy Action Space Linear Function Approximation Sample Efficient Action Value Function

June 24, 2022

Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings
Masatoshi Uehara, Ayush Sekhari, Jason D. Lee, Nathan Kallus, Wen Sun
Reinforcement Learning Markov Decision Process Partially Observable Markov Decision Process Latent Causal Action Value Function

June 22, 2022

PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning
Hanhan Zhou, Tian Lan, Vaneet Aggarwal
Multi Agent Reinforcement Learning Counterfactual Prediction Utility Maximization Action Value Function Value Function Factorization

June 7, 2022

Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structure
Tyler Sam, Yudong Chen, Christina Lee Yu
Optimal Policy Low Rank Inner Structure Intermediate Latent Sample Efficient Reinforcement Learning Action Value Function

May 30, 2022

Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning
Rafael Pina, Varuna De Silva, Joosep Hook, Ahmet Kondoz
Reinforcement Learning Multi Agent Reinforcement Learning Multi Agent Multi Agent Setting Residual Reinforcement Learning Action Value Function Value Function Factorization

March 28, 2022

UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios
Jiajun Chai, Weifan Li, Yuanheng Zhu, Dongbin Zhao, Zhe Ma, Kewu Sun, Jishiyu Ding
Multi Agent Reinforcement Learning Multi Agent System Multi Agent Scenario Multi Agent Network Action Value Function

December 16, 2021

Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigation
Enrico Marchesini, Alessandro Farinelli
Deep Reinforcement Learning Network Programming Navigation Task Action Value Function Multi Robot Reinforcement Learning

Action Value Function

Papers

Operator World Models for Reinforcement Learning

Reinforcement Learning in Categorical Cybernetics

Data-Driven Knowledge Transfer in Batch $Q^*$ Learning

Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning

Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent

ConcaveQ: Non-Monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning

Probabilistic Inference in Reinforcement Learning Done Right

Leveraging World Model Disentanglement in Value-Based Multi-Agent Reinforcement Learning

Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery

TVDO: Tchebycheff Value-Decomposition Optimization for Multi-Agent Reinforcement Learning

An Improved Data Augmentation Scheme for Model Predictive Control Policy Approximation

ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning

Linear Reinforcement Learning with Ball Structure Action Space

Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings

PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning

Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structure

Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning

UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios

Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigation