Distributionally Robust Deep Q-Learning

Chung I Lu; Julian Sester; Aijia Zhang

cs.LGMay 25, 2025

Distributionally Robust Deep Q-Learning

Authors: Chung I Lu, Julian Sester, Aijia Zhang

Organizations: National University of Singapore, Department of Mathematics, 21 Lower Kent Ridge Road, 119077.

Abstract

We propose a novel distributionally robust $Q$ -learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterised with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S&{P}~500 data. We also establish convergence guarantees for exact and approximate robust fitted $Q$ -iteration, decompose the numerical RDQN error into interpretable components, and discuss extensions to compact continuous action sets.

Figures & tables

\widehat{\mathcal{H}_{\delta}}Q_{{\rm NN}}(\lambda_{i},\theta_{\text{target}};x_{i},a_{i})=-\lambda_{i}\varepsilon-\lambda_{i}\delta\left[C_{i}+\log\left(\frac{1}{N_{\nu}}\sum_{j=1}^{N_{\nu}}e^{\left(E_{i,j}-C_{i}\right)}\right)\right],

Algorithm 1 Robust DQN

Z_{i,j}=-r(x_{i},a_{i},x_{i,j}^{\nu})-\gamma\ \min_{k\in\{1,2\}}Q_{\rm NN}\left(\theta_{\text{target},k};x^{\nu}_{i,j},\pi_{\rm NN}(\phi_{\text{target}};x^{\nu}_{i,j})\right);

Algorithm 2 Robust TD3 (one update)

Panel of Figure 1 — Figure 1: The worst case cumulative distribution function for different values of $\delta$ with $\nu=$ Uniform $(0,1)$

Panel of Figure 2 — Figure 2: The worst case cumulative distribution function for different values of $\delta$ with $\nu=$ Beta $(1,5)$

Figure 3: The expected reward for each action $a_{t+1}$ based on the current state $x_{t}$ and action $a_{t}$ under $\mathbb{P}(x_{t},a_{t})$ . The x-axis is the state $x$ and the y-axis is the expected reward for the action taken.

Panel of Figure 4 — Figure 4: Out-of-sample trajectories of the RDQN configuration with the highest log wealth per annum, $\varepsilon=0.002$ , $\delta=0.00005$ , against the DQN control. To avoid selecting on the outcome, each panel shows the median of the five seeds ranked by log wealth. Within each figure, the top graph shows the wealth trajectory of the agent (blue line) and the S&P 500 index (orange line). The middle graph shows the positions taken and the bottom graph shows the maximum drawdown of the agent (blue line) and the S&P 500 index (orange line).

Panel of Figure 5 — Figure 5: Out-of-sample trajectories of the best performing RTD3 configuration, $\varepsilon=0.04$ , $\delta=0.0005$ , against the TD3 control. To avoid selecting on the outcome, each panel shows the median of the five seeds ranked by log wealth. Within each figure, the top graph shows the wealth trajectory of the agent (blue line), equally weighted (balanced) strategy (orange line) and the equally weighted (buy and hold) strategy (green line). The middle graph shows the positions taken and the bottom graph shows the maximum drawdown of the agent (blue line) and the two baselines.

Explore similar work

Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation
Oct 2, 2025Saptarshi Mandal, Yashaswini Murthy, R. SrikantQ-Learning Offline Reinforcement Learning
ECE, UIUC · ORIE, UT Austin · ECE, CSL, and the director of NCSA at UIUC
Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study
Sep 7, 2026Aayush Patel, Andrzej RuszczyńskiMarkov Decision Processes Robot Navigation
Department of Management Science and Information Systems, Rutgers University
Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning
Jun 8, 2025Yang Xu, Swetha Ganesh, Vaneet AggarwalQ-Learning Soft Actor-Critic
Purdue University, West Lafayette, Indiana, USA 47907

Oct 2, 2025, cs.LGJ/K move · Enter open · S save

Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation

Saptarshi Mandal, Yashaswini Murthy, R. Srikant

ECE, UIUC · ORIE, UT Austin · ECE, CSL, and the director of NCSA at UIUC

Q-Learning Offline Reinforcement Learning Bellman Equation Finite Time Analysis Convergence

Open paper AskPDF arXiv

Distributionally robust reinforcement learning seeks policies that remain effective when the deployment environment differs from the one that generated the training data. We study model-free robust Q-learning with

χ^2

uncertainty sets and linear function approximation, using data from a single trajectory of an unknown nominal MDP. Evaluating the

χ^2

robust Bellman target introduces the square root of a conditional second moment, which cannot be estimated unbiasedly from one transition, while the projected robust Bellman operator need not be contractive. We address these obstacles through a variational reformulation of the robust Bellman target and a blockwise frozen-target scheme, and establish a finite-time error bound relative to the optimal robust Q-function for every

γ\in(0,1)

. A neural-network experiment illustrates how the variational target can be used in a continuous-state nonlinear-control task.

Abstract

Explore similar work

Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation

Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study

Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning