cs.LGSep 18, 2026

Deep Reinforcement Learning with Buffered Quantile Objectives

Authors: Mohammad Alipour-vaezi, Sajad Khodadadian

Organizations: Virginia Tech

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision-making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.

Figures & tables

Figure 1: Learning performance on the asset-selling environment for τ∈{0.1,0.9}\tau\in\{0.1,0.9\} . The top row reports the cumulative τ\tau -quantile policy gap, while the bottom row reports cumulative regret under the expected-return criterion. Curves show the mean over three independent seeds; shaded regions denote pointwise 95%95\% Student- tt intervals. Lower values indicate better performance.
Figure 2: Selected final Deep–BQRL policies in the asset-selling environment for τ∈{0.1,0.5,0.9}\tau\in\{0.1,0.5,0.9\} , evaluated at Ch=0C_{h}=0 . Blue cells indicate Continue\mathrm{Continue} and gray cells indicate Stop\mathrm{Stop} . The panels show individual runs with seeds 1004210042 , 4242 , and 2004220042 , respectively, rather than policies averaged across runs. The absorbing post-sale state is omitted.
Figure 3: Episodic reward on the slippery FrozenLake environment for τ∈{0.1,0.5,0.9}\tau\in\{0.1,0.5,0.9\} . Curves report the 50-episode moving-average reward averaged over three independent seeds; shaded regions denote pointwise 95% Student- tt confidence intervals.
Figure 4: Learning performance on the slippery FrozenLake environment for τ=0.5\tau=0.5 . The top panel reports the cumulative τ\tau -quantile policy gap, while the bottom panel reports cumulative regret under the expected-return criterion. Lower values indicate better performance.
More in this topic

Risk-Sensitive RL

Risk-sensitive reinforcement learning optimizes policies for downside risk, tail outcomes, uncertainty, or safety constraints beyond expected reward.

See all 85 papers

Explore similar work

CardsList
Aug 23, 2026

Risk-Sensitive Reinforcement Learning with Smoothed Quantile Objectives

Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields, including healthcare, finance, etc. A popular approach to incorporate risk sensitivity is to optimize a specific quantile of the cumulative reward distribution. However, exact quantile objectives are non-smooth and can change abruptly under small perturbations of the return distribution, making them difficult to optimize reliably when the transition model must be learned from data. Motivated by this instability, we develop UCB-BQRL, a model-based optimistic learning algorithm that maintains confidence sets for the transition kernel and plans using a lower-buffered quantile criterion. The buffered criterion smooths the exact quantile objective by averaging nearby lower quantiles, thereby improving stability under transition-estimation error. To compute the buffered-quantile policy at each episode, we introduce EVI-BQ, an exact dynamic-programming procedure. We establish a high-probability regret bound for UCB-BQRL, which up to logarithmic factors scales as O(eτ/ρτ+H2SAT)\mathcal{O}(\mathrm{e}^{τ/ρ_τ}+H^2\sqrt{SAT}), where ρτρ_τ is denoted as the root-level left-plateau threshold, which is a problem-dependent constant. Further, we establish an information-theoretic lower bound of Ω(H/ρτAT)Ω(H/ρ_τ\sqrt{AT}) for the regret of any algorithm dealing with a quantile objective function. Finally, we prove that the exact point-quantile evaluation and exact lower-buffered quantile evaluation are PP-hard under polynomial-time Turing reductions, even for a fixed policy in a two-state, one-action finite-horizon MDP.

Mohammad Alipour-Vaezi, Huaiyang Zhong, Sajad Khodadadian

Grado Department of Industrial & Systems Engineering, Virginia Tech, Blacksburg, VA 24061, USA

Risk-Sensitive RLRegret Minimization in RLDistributional RLModel-Based RLUCB Algorithms
AskPDF
May 5, 2026

Quantile Geometry Regularization for Distributional Reinforcement Learning

Quantile-based distributional reinforcement learning methods learn return distributions through sampled quantile regression, but their bootstrapped target quantiles may induce distorted or degenerate distribution estimates. We propose Robust Quantile-based Implicit Quantile Networks (RQIQN), a lightweight Wasserstein distributionally robust enhancement boosted from a quantile estimation perspective. We first reinterpret a snapshot of IQN loss as a collection of local empirical quantile estimation problems over sampled current fractions. We then robustify each local slot with a Wasserstein distributionally robust quantile estimation formulation, yielding a closed-form, fraction-dependent correction to the Bellman target. This correction directly addresses distributional degeneration: its median antisymmetry preserves the risk-neutral quantile average, while its monotonicity enlarges upper-lower quantile gaps and counteracts collapsed distributional spread. RQIQN thus regularizes quantile geometry without changing the underlying value objective or requiring additional sample set reconstruction. Finally, we empirically show that the proposed RQIQN outperforms other existing quantile-based distributional reinforcement learning algorithms in risk-sensitive navigation and Atari games.

Zhaofan Zhang, Minghao Yang, Rufeng Chen +2

Information Hub, AI Thrust Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China

Distributional RLQuantile RegressionRisk-Sensitive RLDistributionally Robust OptimizationDistributionally Robust RL
AskPDF
May 25, 2025

Distributionally Robust Deep Q-Learning

We propose a novel distributionally robust QQ-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterised with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S&{P}~500 data. We also establish convergence guarantees for exact and approximate robust fitted QQ-iteration, decompose the numerical RDQN error into interpretable components, and discuss extensions to compact continuous action sets.

Chung I Lu, Julian Sester, Aijia Zhang

National University of Singapore, Department of Mathematics, 21 Lower Kent Ridge Road, 119077.

Robust Markov Decision ProcessesDeep Q-LearningDistributionally Robust OptimizationDistributionally Robust RLRobust RLPolicy Learning
AskPDF