Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields, including healthcare, finance, etc. A popular approach to incorporate risk sensitivity is to optimize a specific quantile of the cumulative reward distribution. However, exact quantile objectives are non-smooth and can change abruptly under small perturbations of the return distribution, making them difficult to optimize reliably when the transition model must be learned from data. Motivated by this instability, we develop UCB-BQRL, a model-based optimistic learning algorithm that maintains confidence sets for the transition kernel and plans using a lower-buffered quantile criterion. The buffered criterion smooths the exact quantile objective by averaging nearby lower quantiles, thereby improving stability under transition-estimation error. To compute the buffered-quantile policy at each episode, we introduce EVI-BQ, an exact dynamic-programming procedure. We establish a high-probability regret bound for UCB-BQRL, which up to logarithmic factors scales as O(eτ/ρτ+H2SAT), where ρτ is denoted as the root-level left-plateau threshold, which is a problem-dependent constant. Further, we establish an information-theoretic lower bound of Ω(H/ρτAT) for the regret of any algorithm dealing with a quantile objective function. Finally, we prove that the exact point-quantile evaluation and exact lower-buffered quantile evaluation are PP-hard under polynomial-time Turing reductions, even for a fixed policy in a two-state, one-action finite-horizon MDP.
Figures & tables
Figure 1: Cumulative expected-return regret RE(k) in the asset-selling experiment when UCB–BQRL is trained with target quantile τ=0.1 . Regret is measured relative to the finite-horizon expected-return-optimal benchmark. Curves show the mean over the three held-out seeds, and shaded regions show the descriptive pointwise variability bands defined in the preceding subsection.
Figure 2: Cumulative numerical τ -quantile policy gap Gτ(k) in the asset-selling experiment for τ=0.5 , measured relative to the approximate-frontier numerical reference. Curves show the mean over the three held-out seeds, and shaded regions show the descriptive pointwise variability bands defined in the preceding subsection.
Figure 3: Cumulative numerical τ -quantile policy gap Gτ(k) in the asset-selling experiment for τ=0.9 , measured relative to the capped-frontier numerical reference. Curves show the mean over the three held-out seeds, and shaded regions show the descriptive pointwise variability bands defined in the preceding subsection.
Figure 4: Selected final UCB–BQRL policies in the asset-selling experiment for τ∈{0.1,0.5,0.9} . For each target quantile, the displayed policy is taken from the held-out run with the smallest final quantile error; buffered-quantile error and seed number are used as tie-breakers. Dark-blue cells indicate Continue and light-gray cells indicate Stop . The displayed policies are selected for structural illustration and should not be interpreted as averages across seeds. The absorbing post-sale state is omitted.
Figure 5: Quantile functions of the root return law in the information-theoretic lower-bound construction. For the distinguished action a⋆ , the return is 0 with probability τ−ρ and H−1 otherwise, so its quantile function jumps at τ−ρ . For any suboptimal action a=a⋆ , the return is 0 with probability τ+ρ and H−1 otherwise, so its quantile function jumps at τ+ρ . Therefore, at the target level τ , the distinguished action has Qτ(Ga⋆)=H−1 , whereas any suboptimal action has Qτ(Ga)=0 . The open and closed circles reflect the left-continuous quantile convention.
Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision-making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.
Risk-sensitive reinforcement learning (RL) addresses decision-making under uncertainty and adverse outcomes. Learning risk-sensitive policies from fixed datasets is important when additional interaction with the environment is costly. Existing work on risk-sensitive RL focuses on online settings or specific risk measures, leaving a general theoretical understanding of its offline setting underdeveloped. In this work, we investigate efficient offline risk-sensitive RL using the optimized certainty equivalent (OCE), a general class of risk measures. We consider two formulations: dynamic-OCE RL, which applies OCE recursively at each step, and static-OCE RL, which applies a single OCE to the cumulative rewards. Under the tabular setting, we develop pessimistic value-iteration algorithms for both formulations and establish tight sublinear suboptimality bounds, explicitly capturing how risk sensitivity, encoded by utility functions, affects learning difficulty. Moreover, we establish minimax suboptimality lower bounds to characterize the problem's fundamental limits. To handle large state spaces, we further develop pessimistic least-squares value-iteration algorithms that incorporate the general OCE risk criterion into linear function approximation. We establish utility-dependent sublinear guarantees with explicit dependence on the feature dimension d. To our knowledge, we provide the first offline algorithms along with the first upper and lower suboptimality bounds for both dynamic- and static-OCE RL.
Hanzhen Wang, Boxiang Lyu, Zhongxiang Dai +3
City University of Hong Kong Hong Kong, China · University of Chicago Chicago, IL, USA · The Chinese University of Hong Kong, Shenzhen Shenzhen, Guangdong, China +1
We consider the policy evaluation and control in a finite horizon reinforcement learning (RL) setting under an expectile-based objective. First, we derive the mean-squared error (MSE) and concentration bounds for the classic estimator of expectiles based on independent and identically distributed (i.i.d.) samples. To the best of our knowledge, expectiles have not been analyzed in the non-asymptotic regime and the bounds we derive may be of independent interest. Next, we analyze a Monte Carlo type estimator of expectile of the Markov chain underlying a given policy. We derive upper bounds that hold in expectation as well as with high probability for this estimator. Further, we show the order-optimality of our estimator by deriving a lower bound on expectile-based policy evaluation. For the problem of control, we adopt a policy gradient approach and derive a policy gradient theorem for expectiles. Using this result, we propose a gradient estimator with a O(1/m) mean-squared error bounds, where m is the number of trajectories. Further, under standard assumptions for policy gradient-type algorithms, we establish smoothness of the expectile-sensitive objective, in turn leading to stationary convergence rate bounds for the overall risk-sensitive policy gradient algorithm that we propose. Finally, we conduct numerical experiments to show the utility of expectiles on popular RL benchmarks.