q-fin.CPJul 5, 2026

Deep Learning for Dynamic Programming with Recursive Utility

Authors: Xianhua Peng, Wu Guo

Abstract

We propose the first deep learning algorithm, the Certainty Equivalent Learning (CEL) algorithm, for solving high-dimensional discrete-time dynamic programming problems with recursive utility. Dynamic programming with recursive utility is numerically challenging because the recursive utility does not have an explicit representation and the Bellman equation contains a certainty equivalent that is difficult to evaluate. The CEL algorithm learns this certainty-equivalent value directly with neural networks and jointly approximates value functions, policy functions, and certainty-equivalent functions. The CEL algorithm is mesh-free and simulation-based, allowing high-dimensional state and control spaces, and does not rely on Euler equations, first-order conditions, or differentiability of the state transition function. The CEL algorithm also works for dynamic programming problems with expected utility as expected utility is a special case of recursive utility. We apply the CEL to discounted linear exponential quadratic Gaussian control, small-noise robust control, Epstein-Zin DSGE, and multivariate strategic asset allocation problems. Compared with closed-form and VFI-based benchmarks, the CEL delivers accurate value and policy approximations, remains effective in high-dimensional problems, achieves accuracy comparable to VFI in the small-noise robust-control case, and produces out-of-sample Bellman errors and Euler or first-order residuals that are in the range from 1.0e-4 to 1.0e-3 for most problems.

Explore similar work

CardsList
Jun 10, 2026q-fin.CP

Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems

In this paper, we develop a continuous-time model-free reinforcement learning algorithm to learn deterministic equilibrium policies in general time-inconsistent control problems. Utilizing the extended Hamilton-Jacobi-Bellman system, we recast the original time-inconsistent problem into an equivalent two-stage problem. In the first stage, for given auxiliary functions, we employ the deterministic policy gradient approach to learn an optimal policy in an auxiliary time-consistent control problem. In the second stage, given the updated policy, we exploit the inner fixed point iterations and some martingale characterizations to learn the auxiliary functions. As a theoretical contribution, we provide some mild model assumptions and establish the convergence of inner fixed point iterations. By repeating this actor-critic style of iterations across two stages, our algorithm aims to learn the equilibrium under different sources of time-inconsistency in a unified manner. The superior effectiveness of the proposed algorithm are illustrated in two classical financial applications with time-inconsistency: mean-variance portfolio management and optimal tracking portfolio under non-exponential discounting.
Xin Guo, Yijie Huang, Xiang Yu
Optimal PoliciesNon-Exponential DiscountingPolicy GradientOptimal ControlEquilibriumPortfolio Optimization
AskPDF
May 8, 2026cs.LG

Actor-Critic Algorithm for Dynamic Expectile and CVaR

Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, a model-free off-policy actor-critic algorithm is constructed. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.
Yudong Luo, Erick Delage
Conditional-Value-At-RiskSoft Actor-CriticModel-Free Reinforcement LearningAI Risk AssessmentSequential Learning
AskPDF
May 20, 2026cs.LG

On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalents

We study risk-sensitive reinforcement learning in finite discounted MDPs, where a generative model of the MDP is assumed to be available. We consider a family or risk measures called the optimized certainty equivalent (OCE), which includes important risk measures such as entropic risk, CVaR, and mean-variance. Our focus is on the sample complexities of learning the optimal state-action value function (value learning) and an optimal policy (policy learning) under recursive OCE. We provide an exact characterization of utility functions uu for which the corresponding OCE defines an objective that is PAC-learnable. We analyze a simple model-based approach and derive PAC sample complexity bounds. We establish that whenever uu does not have full domain dom(u)≠R\text{dom}(u)\neq \mathbb{R}, the corresponding problem is not PAC-learnable. Finally, we establish corresponding lower bounds for both value and policy learning, demonstrating tightness in the size SASA of state-action space, and for a more restricted class of utilities, we derive lower bounds that makes the dependence on the effective horizon 11−γ\frac{1}{1-γ} explicit. Specifically, for CVaRτ\text{CVaR}_τ we show that the correct dependence on ττ is 1τ2\frac{1}{τ^2}, thus improving by a factor of 1τ\frac{1}τ over state-of-the-art although our bound has a suboptimal dependence on 11−γ\frac{1}{1-γ}.
Oliver Mortensen, Mohammad Sadegh Talebi
Value FunctionsMarkov Decision ProcessesOptimal Sample ComplexitySample Complexity
AskPDF