Risk-averse learning with delayed feedback

Siyi Wang; Zifan Wang; Karl Henrik Johansson; Sandra Hirche

cs.LGSep 25, 2024

Risk-averse learning with delayed feedback

Authors: Siyi Wang, Zifan Wang, Karl Henrik Johansson, Sandra Hirche

Organizations: Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 10044 Stockholm, Sweden · Chair of Information-oriented Control (ITR), Technical University of Munich, Germany

Abstract

In real-world scenarios, risk-averse learning is valuable for mitigating potential adverse outcomes. However, the delayed feedback makes it challenging to assess and manage risk effectively. In this paper, we investigate risk-averse learning using Conditional Value at Risk (CVaR) as risk measure, while incorporating feedback with random but bounded delays. We develop two risk-averse learning algorithms that rely on one-point and two-point zeroth-order optimization approaches, respectively. The dynamic regrets of the algorithms are analyzed in terms of the cumulative delay and the number of total samplings. In the absence of delay, the regret bounds match the established bounds of zeroth-order stochastic gradient methods for risk-averse learning. Furthermore, the two-point risk-averse learning outperforms the one-point algorithm by achieving a smaller regret bound. We provide numerical experiments on a dynamic pricing problem to demonstrate the performance of the algorithms.

Figures & tables

Panel of Figure 1 — Fig. 1: An example illustrates the delay model: the learner does not receive feedback at slot $t-1$ , i.e., $\mathcal{L}_{t-1}=\emptyset$ , and receives two feedbacks at slot $t$ , which are generated at slots $t-2$ and $t$ , respectively, i.e., $\mathcal{L}_{t}=\{J_{s_{1|t}},u_{s_{1|t}},J_{s_{2|t}},u_{s_{2|t}}\}$ . with $s_{1|t}=t-2$ and $s_{2|t}=t$ .

Panel of Figure 2 — Fig. 2: Real-to-virtual mapping.

Fig. 3: Price and occupancy trajectories of the one- and two-point methods under the step-changing target ( 51 ). The optimal price and target occupancy are shown for reference.

Fig. 4: CVaR values and dynamic regret of the one- and two-point methods under the step-changing target ( 51 ). The optimal CVaR value is shown for reference.

Fig. 5: Price, occupancy, and dynamic regret of the one- and two-point methods under the smoothly varying target

r_{t}^{d}=0.7+0.05\cos(t/80)

. The optimal price and target occupancy are shown for reference.

Fig. 6: Effect of the maximum delay on the two-point method: price trajectories and dynamic regret for

\bar{d}\in\{0,20,50\}

Fig. 7: Effect of the batch size on the two-point method: price trajectories and dynamic regret for the constant batch sizes

m_{t}\in\{1,4,16\}

Fig. 8: Price trajectories and dynamic regret under

\bar{d}=20

for the delay-aware first-order ( [ 33 ] ), one-point (Alg. 1 ), and two-point methods (Alg. 2 ) and the delay-neglecting one- and two-point variants. The optimal price is shown for reference.

Fig. 9: Risk–performance tradeoff under bounded rare-event uncertainty: common expected loss

\mathrm{CVaR}_{1}

, worst-decile loss

\mathrm{CVaR}_{0.1}

, and severe over-occupancy probability

\Pr(r_{t}>r_{t}^{d}+0.1)

for policies trained with

\alpha\in\{0.1,0.5,1\}

Dynamic Regret Minimization

Dynamic regret minimization studies how online learners can track changing decision-makers while limiting cumulative loss.

See all 46 papers

Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning
Aug 5, 2026Yifan Wu, Junjie Lei, Wenjie HuangReinforcement Learning Q-Learning
Department of Data and Systems Engineering The University of Hong Kong Hong Kong, Hong Kong SAR
Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
Feb 3, 2026Aneri Muni, Vincent Taboga, Esther Derman +2Markov Decision Processes Q-Learning
Universit´e de Montr´eal, Quebec, Canada · Mila - Quebec AI Institute, Quebec, Canada · GERAD & Department of Decision Sciences, HEC Montr´eal, Quebec, Canada
Actor-Critic Algorithm for Dynamic Expectile and CVaR
May 8, 2026Yudong Luo, Erick DelageReinforcement Learning Risk-Sensitive RL
1GERAD & Department of Decision Sciences, HEC Montr´eal, Canada · 2Mila - Quebec AI Institute, Canada

Aug 5, 2026J/K move · Enter open · S save

Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning

Yifan Wu, Junjie Lei, Wenjie Huang

Department of Data and Systems Engineering The University of Hong Kong Hong Kong, Hong Kong SAR

Reinforcement Learning Q-Learning Risk-Sensitive RL Conditional Value-at-Risk Finance

Open paper AskPDF arXiv

Risk-aware Q-learning (RaQL) provides a model-free, two-timescale estimator for dynamic risk objectives, but its finite-budget behavior remains fragile: fixed inner-loop hyperparameters can produce unstable value estimates, persistent Bellman residuals, and inefficient sample reuse. This paper proposes an adaptive training controller for Conditional Value-at-Risk (CVaR) RaQL and evaluates it on a daily Bitcoin trading task. The controller preserves the original CVaR estimator and Bellman fixed point; instead, it redesigns the training procedure through six coordinated mechanisms: per-cell inner-step sizing, outer-rate-matched decay synchronization, a short early correction for the VaR-like inner variable, a coverage-first-then-greedy sample allocation rule, progressive suffix aggregation of mature inner estimates, and data-driven calibration of key scales from online-observable quantities. Across 20 random seeds and 856,000 inner-transition samples, the controller reduces the mean empirical CVaR Bellman residual by approximately 85% relative to the fixed-parameter baseline (MeanBEQ: 1.2202 to 0.1854; MeanBEV: 1.1624 to 0.0535) and maintains stability across CVaR levels, discount factors, and training budgets. On the chronological out-of-sample test set, the learned policy attains a Sharpe ratio of 0.9281 with a maximum drawdown of 6.46% after transaction costs. Although buy-and-hold yields a higher cumulative return (35.43% vs. 23.61%), the adaptive policy achieves far lower volatility (9.57% vs. 47.93%), drawdown, and CVaR loss. These results demonstrate that adaptive finite-budget training design, applied solely to the training procedure without altering the risk objective, can materially improve the reliability and risk-adjusted performance of risk-aware Q-learning in financial applications.

Abstract

Dynamic Regret Minimization

Explore similar work

Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning

Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity

Actor-Critic Algorithm for Dynamic Expectile and CVaR