Risk Sensitive Reinforcement Learning

Risk-sensitive reinforcement learning (RL) aims to develop agents that make optimal decisions while considering and mitigating risk, unlike traditional RL which focuses solely on maximizing expected reward. Current research emphasizes efficient algorithms for various risk measures (e.g., Conditional Value-at-Risk, entropic risk), often within specific model frameworks like Markov Decision Processes (MDPs) and incorporating techniques from optimal transport and distributional RL. This field is crucial for deploying RL in high-stakes applications where safety and robustness are paramount, impacting areas such as finance, healthcare, and robotics by enabling more reliable and safer autonomous systems. Recent work focuses on improving sample efficiency and providing theoretical guarantees for algorithm convergence and regret bounds.

Papers

June 4, 2023

Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures
Hao Liang, Zhi-quan Luo
Regret Bound Risk Sensitive Episodic Markov Decision Process Risk Sensitive Reinforcement Learning Lipschitz Operator Risk Measure Spectral Risk

February 7, 2023

Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR
Kaiwen Wang, Nathan Kallus, Wen Sun
Optimal Regret Minimax Regret Risk Sensitive Reinforcement Learning Multi Arm Bandit Conditional Value at Risk

January 30, 2023

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents
Wenhao Xu, Xuefeng Gao, Xuedong He
Markov Decision Process Regret Bound Recursive Algorithm Risk Sensitive Reinforcement Learning Tabular Markov Decision Process Optimal Dependence Certainty Equivalent Entropic Risk

January 26, 2023

On the Global Convergence of Risk-Averse Policy Gradient Methods with Expected Conditional Risk Measures
Xian Yu, Lei Ying
Policy Gradient Global Convergence Risk Sensitive Reinforcement Learning Risk Measure Competing Risk Risk Averse Policy

December 18, 2022

Risk-Sensitive Reinforcement Learning with Exponential Criteria
Erfaun Noorani, Christos Mavridis, John Baras
Reinforcement Learning Actor Critic Algorithm Risk Sensitive Reinforcement Learning Robust Deep Reinforcement Learning

November 30, 2022

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
Marc Rigter, Bruno Lacerda, Nick Hawes
Offline Reinforcement Learning Epistemic Uncertainty Risk Sensitive High Quality Risk Description Model Based Offline Reinforcement Learning Risk Sensitive Reinforcement Learning High Epistemic Uncertainty

November 19, 2022

Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Design
Yuhao Ding, Ming Jin, Javad Lavaei
Markov Decision Process Separation Performance Risk Sensitive Reinforcement Learning Non Stationary Reinforcement Learning Optimal Dynamic Regret Entropic Risk Adaptive Detection

October 25, 2022

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds
Hao Liang, Zhi-Quan Luo
Regret Bound Distributional Reinforcement Learning Regret Analysis Risk Sensitive Reinforcement Learning

October 11, 2022

Regret Bounds for Risk-Sensitive Reinforcement Learning
O. Bastani, Y. J. Ma, E. Shen, W. Xu
Reinforcement Learning Regret Bound Risk Sensitive Risk Sensitive Reinforcement Learning CVaR Objective

September 15, 2022

IoT-Aerial Base Station Task Offloading with Risk-Sensitive Reinforcement Learning for Smart Agriculture
Turgay Pamuklu, Anne Catherine Nguyen, Aisha Syed, W. Sean Kennedy, Melike Erol-Kantarci
Risk Sensitive Reinforcement Learning Smart Agriculture Base Station

June 29, 2022

Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning
Anthony Coache, Sebastian Jaimungal, Álvaro Cartea
Deep Reinforcement Learning Actor Critic Algorithm Conditional Reasoning Risk Sensitive Reinforcement Learning Elicitation Method Spectral Risk Dynamic Risk

June 6, 2022

Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path
Yihan Du, Siwei Wang, Longbo Huang
Regret Minimization Worst Case Risk Sensitive Reinforcement Learning CVaR Objective

March 7, 2022

Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learning
Yingjie Fei, Ruitu Xu
Filling Gap Risk Sensitive Reinforcement Learning Gap Dependent Sub Optimality Gap Logarithmic Bayes Regret

February 22, 2022

A policy gradient approach for optimization of smooth risk measures
Nithia Vijayan, Prashanth L. A
Optimization Purpose Policy Gradient Episodic Markov Decision Process Risk Sensitive Reinforcement Learning

November 6, 2021

Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning
Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang
Regret Bound Risk Sensitive Risk Sensitive Reinforcement Learning Bellman Equation

Risk Sensitive Reinforcement Learning

Papers

Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures

Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents

On the Global Convergence of Risk-Averse Policy Gradient Methods with Expected Conditional Risk Measures

Risk-Sensitive Reinforcement Learning with Exponential Criteria

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Design

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds

Regret Bounds for Risk-Sensitive Reinforcement Learning

IoT-Aerial Base Station Task Offloading with Risk-Sensitive Reinforcement Learning for Smart Agriculture

Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning

Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path

Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learning

A policy gradient approach for optimization of smooth risk measures

Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning