Near Optimal Policy

Near-optimal policy research in reinforcement learning focuses on developing efficient algorithms that find policies achieving performance very close to the theoretical optimum, addressing the challenges of large state spaces and limited data. Current research emphasizes algorithms leveraging linear function approximation, policy gradients (often incorporating optimistic exploration strategies), and techniques like Whittle indices or primal-dual optimization, sometimes within specific model frameworks such as Markov Decision Processes (MDPs) or restless multi-armed bandits. These advancements improve sample efficiency and computational tractability, leading to more practical applications in diverse fields like robotics, resource management, and personalized decision-making systems.

Papers

September 7, 2024

Sample and Oracle Efficient Reinforcement Learning for MDPs with Linearly-Realizable Value Functions
Zakaria Mhammedi
Markov Decision Process Near Optimal Policy Plausible Learning Linear Markov Decision Process

August 29, 2024

Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form
Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai, Kenta Hoshino, Yohei Hosoe, Kazumi Kasaura, Masashi Hamaya, Paavo Parmas, Yutaka Matsuo
Markov Decision Process Policy Gradient Near Optimal Policy Policy Identification

August 19, 2024

GINO-Q: Learning an Asymptotically Optimal Index Policy for Restless Multi-armed Bandits
Gongpu Chen, Soung Chang Liew, Deniz Gunduz
Neural Operator Stochastic Approximation Near Optimal Policy Traditional Reinforcement Learning Restless Multi Armed Bandit Index Policy

June 7, 2024

Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning
Subhojyoti Mukherjee, Josiah P. Hanna, Qiaomin Xie, Robert Nowak
Multi Task Decision Transformer Near Optimal Policy Reward Prediction Contextual Bandit Setting Structured Bandit

May 28, 2024

An approach to improve agent learning via guaranteeing goal reaching in all episodes
Pavel Osinenko, Grigory Yaremenko, Georgiy Malaniya, Anton Bolychev
Reinforcement Learning Markov Decision Process Constructive Approach Near Optimal Policy Pseudo Goal Uncertain Episode Length Agent Learning Reward Maximizing

May 25, 2024

Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, Marek Cygan
Reinforcement Learning Near Optimal Policy Optimistic Exploration Positive Scaling

May 23, 2024

Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time
Jeremy McMahan
Constrained Reinforcement Learning Near Optimal Policy Deterministic Policy Approximate Dynamic Programming Polynomial Time Approximation

March 8, 2024

Switching the Loss Reduces the Cost in Batch Reinforcement Learning
Alex Ayoub, Kaiwen Wang, Vincent Liu, Samuel Robertson, James McInerney, Dawen Liang, Nathan Kallus, Csaba Szepesvári
Hidden CoST Interest Loss Near Optimal Policy Batch Reinforcement Learning Fitted Q Iteration Small Loss

February 21, 2024

Dealing with unbounded gradients in stochastic saddle-point optimization
Gergely Neu, Nneka Okolo
Reinforcement Learning Optimization Purpose Near Optimal Policy Stochastic First Order Method Unbounded Gradient Concave Function

January 31, 2024

Behind the Myth of Exploration in Policy Gradients
Adrien Bolland, Gaspard Lambrechts, Damien Ernst
Policy Gradient Optimal Policy Environment Exploration Near Optimal Policy Policy Gradient Algorithm Exploration Strategy Many Myth Effective Reinforcement Learning

January 7, 2024

Quantifying stability of non-power-seeking in artificial agents
Evan Ryan Gunter, Yevgeny Liokumovich, Victoria Krakovna
Markov Decision Process Core Stability AI Agent Artificial Agent Near Optimal Policy Power Seeking

December 16, 2023

Advancing RAN Slicing with Offline Reinforcement Learning
Kun Yang, Shu-ping Yeh, Menglei Zhang, Jerry Sydir, Jing Yang, Cong Shen
Reinforcement Learning Offline Reinforcement Learning Near Optimal Policy Radio Resource Management Radio Access Network Slicing

November 26, 2023

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation
Heyang Zhao, Jiafan He, Quanquan Gu
Reinforcement Learning Near Optimal Policy Minimax Optimal Regret

November 22, 2023

Risk-sensitive Markov Decision Process and Learning under General Utility Functions
Zhengqi Wu, Renyuan Xu
Reinforcement Learning LeArning Abstract Markov Decision Process Near Optimal Policy Risk Sensitive Reinforcement Learning Utility Function Uncertainty Aware Exploration

October 20, 2023

Provable Benefits of Multi-task RL under Non-Markovian Decision Making Processes
Ruiquan Huang, Yuan Cheng, Jing Yang, Vincent Tan, Yingbin Liang
Markov Decision Process Multi Task Learning Multi Task Near Optimal Policy Multi Task Reinforcement Learning Provable Benefit

October 16, 2023

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms
Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, Tuo Zhao
Multi Agent Reinforcement Learning Robust Optimization Near Optimal Policy Adversarial Regularization Stable Algorithm

October 11, 2023

Prospective Side Information for Latent MDPs
Jeongyeol Kwon, Yonathan Efroni, Shie Mannor, Constantine Caramanis
Markov Decision Process O$ Regret Near Optimal Policy Side Information Partially Observed Markov Decision Process Latent MDPs Latent Markov Decision Process

September 29, 2023

Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
Xiaoyu Wen, Xudong Yu, Rui Yang, Haoyuan Chen, Chenjia Bai, Zhen Wang
Reinforcement Learning High Uncertainty Anticipation Offline Reinforcement Learning Algorithm Near Optimal Policy Policy Improvement Robust Offline Reinforcement Learning Continual Offline Reinforcement Learning

September 27, 2023

Importance-Weighted Offline Learning Done Right
Germano Gabbianelli, Gergely Neu, Matteo Papini
Contextual Bandit Near Optimal Policy Optimal Behavior Offline Policy Implicit Exploration

September 18, 2023

Exploring and Learning in Sparse Linear MDPs without Computationally Intractable Oracles
Noah Golowich, Ankur Moitra, Dhruv Rohatgi
LeArning Abstract Computational Approach Near Optimal Policy Linear Markov Decision Process Computational Learning

Near Optimal Policy

Papers

Sample and Oracle Efficient Reinforcement Learning for MDPs with Linearly-Realizable Value Functions

Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form

GINO-Q: Learning an Asymptotically Optimal Index Policy for Restless Multi-armed Bandits

Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning

An approach to improve agent learning via guaranteeing goal reaching in all episodes

Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control

Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time

Switching the Loss Reduces the Cost in Batch Reinforcement Learning

Dealing with unbounded gradients in stochastic saddle-point optimization

Behind the Myth of Exploration in Policy Gradients

Quantifying stability of non-power-seeking in artificial agents

Advancing RAN Slicing with Offline Reinforcement Learning

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation

Risk-sensitive Markov Decision Process and Learning under General Utility Functions

Provable Benefits of Multi-task RL under Non-Markovian Decision Making Processes

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

Prospective Side Information for Latent MDPs

Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness

Importance-Weighted Offline Learning Done Right

Exploring and Learning in Sparse Linear MDPs without Computationally Intractable Oracles