Certainty Equivalent

Certainty equivalence (CE) is a control and reinforcement learning approach that uses an estimated model of a system to determine optimal actions, treating the estimate as if it were the true model. Current research focuses on improving CE's performance and theoretical understanding, particularly through refined algorithms like optimistic meta-algorithms and state-action-specific regularization techniques, and analyzing its regret bounds in various settings, including Markov Decision Processes and linear systems. This work is significant because it addresses challenges in adaptive control and risk-sensitive decision-making under uncertainty, with implications for diverse applications ranging from robotics and finance to resource management and healthcare.

Papers

May 31, 2024

Concentration Bounds for Optimized Certainty Equivalent Risk Estimation
Ayon Ghosh, L. A. Prashanth, Krishna Jagannathan
Average Approximation Risk Aware Concentration Bound Certainty Equivalent

March 10, 2024

Risk-Sensitive RL with Optimized Certainty Equivalents via Reduction to Standard RL
Kaiwen Wang, Dawen Liang, Nathan Kallus, Wen Sun
Policy OpTimization Risk Sensitive Meta Reinforcement Learning Constructive Reduction Human Like RL Risk Sensitive Reinforcement Learning Certainty Equivalent

June 20, 2023

The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning
Sarah Rathnam, Sonali Parbhoo, Weiwei Pan, Susan A. Murphy, Finale Doshi-Velez
Optimal Policy State Action Pair Regularization Penalty Certainty Equivalent

May 26, 2023

Finite Time Regret Bounds for Minimum Variance Control of Autoregressive Systems with Exogenous Inputs
Rahul Singh, Akshay Mete, Avik Kar, P. R. Kumar
Autoregressive Model Adaptive Controller Exogenous Variable Finite Time Regret Certainty Equivalent

April 2, 2023

Stability Bounds for Learning-Based Adaptive Control of Discrete-Time Multi-Dimensional Stochastic Linear Systems with Input Constraints
Seth Siriya, Jingge Zhu, Dragan Nešić, Ye Pu
High Dimensional Discrete Time Input Constraint Stochastic Disturbance Certainty Equivalent Learning Based Adaptive

January 30, 2023

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents
Wenhao Xu, Xuefeng Gao, Xuedong He
Markov Decision Process Regret Bound Recursive Algorithm Risk Sensitive Reinforcement Learning Tabular Markov Decision Process Optimal Dependence Entropic Risk Certainty Equivalent

December 15, 2022

Rollout Algorithms and Approximate Dynamic Programming for Bayesian Optimization and Sequential Estimation
Dimitri Bertsekas
Bayesian Optimization Average Approximation Dynamic Programming Approximate Dynamic Programming Model Rollouts Optimal Measurement Sequential Estimation Certainty Equivalent

September 15, 2022

Learning-Based Adaptive Control for Stochastic Linear Systems with Input Constraints
Seth Siriya, Jingge Zhu, Dragan Nešić, Ye Pu
Adaptive Control Linear System Input Constraint Certainty Equivalent

June 9, 2022

Regret Analysis of Certainty Equivalence Policies in Continuous-Time Linear-Quadratic Systems
Mohamad Kazem Shirani Faradonbeh
Reinforcement Learning Regret Bound Regret Analysis Ubiquitous Intelligence Certainty Equivalent

February 4, 2022

Learning a Discrete Set of Optimal Allocation Rules in a Queueing System with Unknown Service Rate
Saghar Adler, Mehrdad Moharrami, Vijay Subramanian
Regret Bound Discrete Space Finite Time Regret Efficient Allocation Certainty Equivalent Power Dispatch Contextual Stochastic Block Model

November 13, 2021

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds
Yahya Sattar, Zhe Du, Davoud Ataee Tarzanagh, Laura Balzano, Necmiye Ozay, Samet Oymak
Dynamical System Sample Complexity Person Identification Regret Bound Adaptive Control Markov Jump Certainty Equivalent

Certainty Equivalent

Papers

Concentration Bounds for Optimized Certainty Equivalent Risk Estimation

Risk-Sensitive RL with Optimized Certainty Equivalents via Reduction to Standard RL

The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning

Finite Time Regret Bounds for Minimum Variance Control of Autoregressive Systems with Exogenous Inputs

Stability Bounds for Learning-Based Adaptive Control of Discrete-Time Multi-Dimensional Stochastic Linear Systems with Input Constraints

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents

Rollout Algorithms and Approximate Dynamic Programming for Bayesian Optimization and Sequential Estimation

Learning-Based Adaptive Control for Stochastic Linear Systems with Input Constraints

Regret Analysis of Certainty Equivalence Policies in Continuous-Time Linear-Quadratic Systems

Learning a Discrete Set of Optimal Allocation Rules in a Queueing System with Unknown Service Rate

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds