Heavy Tailed Reward

Heavy-tailed reward distributions, characterized by infrequent but extremely large reward values, pose significant challenges in reinforcement learning (RL) because standard algorithms often fail to converge or perform poorly due to outliers. Current research focuses on developing robust algorithms for various RL settings, including bandits and Markov decision processes, that handle heavy-tailed rewards effectively. These algorithms often employ techniques like truncation, median-of-means estimation, and adaptive Huber regression to mitigate the impact of outliers and achieve near-optimal performance. This research is crucial for advancing the applicability of RL to real-world scenarios where heavy-tailed rewards are common, such as finance and online advertising.

Papers

July 19, 2024

Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso
Reinforcement Learning Optimal Policy Reward Model Reinforcement Learning From Human Feedback Heavy Tailed KL Divergence Heavy Tailed Reward Extinction Risk

April 26, 2024

Low-rank Matrix Bandits with Heavy-tailed Rewards
Yue Kang, Cho-Jui Hsieh, Thomas C. M. Lee
Low Rank Sub Gaussian Heavy Tailed Reward

October 28, 2023

October 4, 2023

Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learning
Weidong Liu, Jiyuan Tu, Yichen Zhang, Xi Chen
Reinforcement Learning Scientific Inference Online Estimation Robust Statistic Heavy Tailed Reward Robust Policy Evaluation

June 20, 2023

Provably Robust Temporal Difference Learning for Heavy-Tailed Rewards
Semih Cayci, Atilla Eryilmaz
Stochastic Gradient Temporal Difference Learning Stochastic Reward Heavy Tailed Reward Truly Robust Case

June 12, 2023

Tackling Heavy-Tailed Rewards in Reinforcement Learning with Function Approximation: Minimax Optimal and Instance-Dependent Regret Bounds
Jiayi Huang, Han Zhong, Liwei Wang, Lin F. Yang
Reinforcement Learning Linear Bandit Linear Function Approximation Function Approximation Heavy Tailed Minimax Optimality Instance Dependent Regret Heavy Tailed Noise Heavy Tailed Reward

June 1, 2023

Differentially Private Episodic Reinforcement Learning with Heavy-tailed Rewards
Yulian Wu, Xingyu Zhou, Sayak Ray Chowdhury, Di Wang
Markov Decision Process Heavy Tailed Heavy Tailed Reward Private Reinforcement Learning Heavy Tailed Policy

May 11, 2023

Implicitly normalized forecaster with clipping for linear and non-linear heavy-tailed multi-armed bandits
Yuriy Dorn, Nikita Kornilov, Nikolay Kutuzov, Alexander Nazin, Eduard Gorbunov, Alexander Gasnikov
Heavy Tailed Linear Ordered Data Implicit Language Forecast Utterance Heavy Tailed Reward Adversarial Multi Armed Bandit Learning Based Forecasting

March 9, 2023

Variance-aware robust reinforcement learning with linear function approximation under heavy-tailed rewards
Xiang Li, Qiang Sun
Linear Bandit Linear Function Approximation Robust Reinforcement Learning Huber Loss Heavy Tailed Reward Variance Aware Variance Dependent Regret Online Sequential Decision Making

February 6, 2023

On Private and Robust Bandits
Yulian Wu, Xingyu Zhou, Youming Tao, Di Wang
Differential Privacy Optimal Regret Heavy Tailed Heavy Tailed Reward Robust Multi Armed Bandit

December 29, 2022

Quantile Off-Policy Evaluation via Deep Conditional Generative Learning
Yang Xu, Chengchun Shi, Shikai Luo, Lan Wang, Rui Song
Policy Evaluation Policy Estimation Nuisance Parameter Heavy Tailed Reward Deep Conditional Generative

June 17, 2022

Multiple-Play Stochastic Bandits with Shareable Finite-Capacity Arms
Xuchuang Wang, Hong Xie, John C. S. Lui
Hybrid Reward Heavy Tailed Reward Reward Distribution

December 8, 2011

The Extended UCB Policies for Frequentist Multi-armed Bandit Problems
Keqin Liu, Tianshuo Zheng, Haoran Chen
Multi Armed Bandit Robust Policy Sequential Decision Making UCB Algorithm Heavy Tailed Reward