Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting
Authors: Mehrdad Moghimi, Anthony Coache, Hyejin Ku
Organizations: Dept. of Mathematics and Statistics, York University, Toronto, Canada · Dept. of Mathematics, Imperial College London, London, United Kingdom
Distributional reinforcement learning (RL) is a powerful framework increasingly adopted in safety-critical domains for its ability to optimize risk-sensitive objectives. However, the role of the discount factor is often overlooked, as it is typically treated as a fixed parameter of the Markov decision process or tunable hyperparameter, with little consideration of its effect on the learned policy. In the literature, it is well-known that the discounting function plays a major role in characterizing time preferences of an agent, which an exponential discount factor cannot fully capture. Building on this insight, we propose a novel framework that supports flexible discounting of future rewards and optimization of risk measures in distributional RL. We provide a technical analysis of the optimality of our algorithms, show that our multi-horizon extension fixes issues raised with existing methodologies, and validate the robustness of our methods through extensive experiments. Our results highlight that discounting is a cornerstone in decision-making problems for capturing more expressive temporal and risk preferences profiles, with potential implications for real-world safety-critical applications.
Figures & tables
Figure 1: Impact of Time-Consistency. Our time-consistent method (blue) outperforms the time-inconsistent approach of Fedus et al. (2019) (red) by modeling the non-stationary policy.
Criterion
Discount
T1
T2
T3
T4
T5
T6
Optimal
E
γ=0.8
16.000
17.920
8.192
4.194
6.040
3.006
T2
E
k=1
10.000
8.750
4.167
2.500
4.500
2.917
T1
Entr0.5
γ=0.8
1.386
1.386
4.648
4.194
4.002
2.961
T3
Entr0.5
k=1
1.386
1.386
2.895
2.500
3.229
2.874
T5
Entr1.5
γ=0.8
0.462
0.462
3.739
4.194
3.146
2.877
T4
Entr1.5
k=1
0.462
0.462
2.128
2.500
2.462
2.794
T6
Table 1: Comparison of optimal plans under different criteria and discount functions.
π^t∗={π^t∗πRN∗if t<T′if t≥T′
Algorithm 1 Distributional VI for Infinite-Horizon MDPs with General Discounting
Figure 2: Preference reversals in GBWM. Monte-Carlo probabilities of achieving the two goals for a risk-neutral (left) and risk-sensitive (right) agent, with various discounting (colors) and environment settings (shapes). Standard errors, not included for clarity, are provided in Appendix H.3 .
Figure 3: Relative performance of our Time-Consistent algorithm on 50 Atari games. Bars show percent improvement over the time-inconsistent baseline (3 seeds). Our method outperforms in 39/50 games, with 39.89% mean and 18.14% median gains, highlighting the benefit of time-consistency under hyperbolic discounting.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: From interest rates to discount functions. Comparison of exponential with γ=0.99 (Orange), hyperbolic with k=0.01 (Green), and CIR-based discount factors with parameters a=0.05 , b=0.01 , σ=0.1 , and r0=0.01 (Blue)
Figure 5: Utility functions for common OCE risk measures. Detailed formulations are provided in Table 2 .
Risk Name
Parameter
Formula
Utility function f(x)
Mean
None
E[G]
x
CVaR
τ∈(0,1]
CVaRτ(G)
τ−1(x)−
Mean-CVaR
κ1∈[0,1]
κ1E[G]+(1−κ1)⋅CVaRτ(G)
κ1(x)++κ2(x)−, where
τ∈(0,1]
κ2=τ−1(1−κ1)+κ1
Entropic Risk
β∈(0,∞)
Entrβ(G)
β1(1−exp(−βx))
Mean-Variance
κ∈(0,∞)
E[G]−κ⋅Var(G)
(x−κx2)I{x≤2κ1}
Appendix
Table 2: Common Risk Measures as OCEs.
Portfolio weights
Portfolio number
U.S. Bonds
Int’l Stocks
U.S. Stocks
1
0.9098
0.0225
0.0677
2
0.8500
0.0033
0.1467
3
0.7903
−0.0160
0.2257
4
0.7305
−0.0352
0.3047
5
0.6707
−0.0545
0.3837
Appendix
Table 3: Portfolio weights, where the most conservative (portfolio 1) has an expected return of 0.0526 and volatility of 0.0374 , while the most aggressive (portfolio 15) has an expectation of 0.0886 and standard deviation of 0.1954 .
Method
E[U]
P[yT/2≥100]
P[ψT/2=1]
P[ψT=1]
uT/2(1)=1000 , uT(1)=1000 , T=10
E , γ=1.00
1104(24.54)
0.848(0.017)
0.848(0.017)
0.256(0.008)
E , γ=0.99
1112(11.81)
0.845(0.040)
0.845(0.040)
0.267(0.029)
E , k=0.05
1108(17.49)
0.864(0.020)
0.864(0.020)
0.244(0.037)
uT/2(1)=1000 , uT(1)=2000 , T=10
E , γ=1.00
1782(10.07)
0.855(0.016)
0.237(0.027)
0.772(0.012)
Appendix
Table 4: Evaluation of the policies learned by Algorithm 2 for a risk-neutral agent with cT/2(1)=100 and cT(1)=150 over 10,000 episodes, evaluated for 3 different runs. Standard errors are reported in round brackets.
Method
E[U]
P[yT/2≥100]
P[ψT/2=1]
P[ψT=1]
uT/2(1)=1000 , uT(1)=1000 , T=10
CVaR0.1 , γ=1.00
999(0.036)
0.998(0.0005)
0.998(0.0005)
0.001(0.0004)
CVaR0.1 , k=0.05
999(1.05)
0.994(0.0036)
0.994(0.0036)
0.005(0.0025)
uT/2(1)=1000 , uT(1)=2000 , T=10
CVaR0.1 , γ=1.00
1007(4.15)
0.986(0.0084)
0.986(0.0084)
0.010(0.0061)
CVaR0.1 , k=0.05
1020(18.05)
0.977(0.0202)
0.977(0.0202)
0.022(0.0148)
Appendix
Table 5: Evaluation of the policies learned by Algorithm 2 for a risk-sensitive agent with cT/2(1)=100 and cT(1)=150 over 10,000 episodes, evaluated for 3 different runs. Standard errors are reported in round brackets.
Hyperparameter
Value
Optimizer
Adam
Learning rate
5×10−5
Batch size B
32
Replay buffer size
1,000,000 transitions
Learning starts
50,000 steps
Training frequency
Every 4 environment steps
Appendix
Table 6: Atari hyperparameters used for the multi-horizon QR-DQN experiments.
Figure 6: Relative performance improvement of our Time-Consistent algorithm across 50 Atari games. Each bar shows the percentage improvement in mean return for the Time-Consistent policy over the Time-aware Time-Inconsistent baseline, averaged over 3 seeds per game. The Time-Consistent policy outperforms the baseline in 33 out of 50 games. Across all games, it achieves a mean improvement of 15.73% and a median improvement of 8.02%, demonstrating the benefits of maintaining time-consistency under hyperbolic discounting.
Figure 7: Comparison on Atari 2600. Learning curves over 10 million steps (40 million frames) for 50 Atari games. Our Time-Consistent agent (Blue) is compared against the Time-Inconsistent hyperbolic baseline from Fedus et al. (2019) (Red) and a time-aware variant of this baseline (Green). Shaded regions represent 95% bootstrap confidence intervals across 3 seeds. The results demonstrate that properly handling the non-stationarity of the optimal policy is critical for effective learning with general discount functions. Note that the reported mean returns are computed using the clipped rewards commonly used during training for stability, rather than the unclipped episodic returns.
Figure 8: Relative performance improvement of our Time-Consistent algorithm over exponentially discounted QR-DQN across 50 Atari games. Each bar shows the percentage improvement in mean return for the Time-Consistent policy over the Exponential baseline with γ=0.99 , averaged over 3 seeds per game. The Time-Consistent policy outperforms the baseline in 40 out of 50 games, with a median improvement of 51.3%.
Figure 9: Relative performance improvement of our Time-Consistent algorithm over the Multi-Horizon (largest γ ) agent across Atari games. Each bar shows the percentage improvement in mean return, averaged over 3 seeds per game. The Time-Consistent policy outperforms the baseline in 36 out of 50 games, with a median improvement of 17.0% over the 49 games shown. Enduro is not shown because the percentage is undefined when the baseline return is zero; we count it among the 36 games since the Time-Consistent policy obtains a positive return.
Figure 10: Relative performance improvement of the Multi-Horizon (largest γ ) agent over exponentially discounted QR-DQN across 50 Atari games. Each bar shows the percentage improvement in mean return, averaged over 3 seeds per game. The Multi-Horizon agent outperforms the Exponential baseline in 30 out of 50 games, with a median improvement of 9.6%.
Figure 11: Comparison with exponentially discounted baselines on Atari 2600. Learning curves for 50 Atari games. Our Time-Consistent agent (Blue) is compared against the Multi-Horizon (largest γ ) agent (Orange) and exponentially discounted QR-DQN with γ=0.99 (Purple). Shaded regions represent 95% bootstrap confidence intervals across 3 seeds. Mean returns are computed using the clipped rewards used during training.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Discounted exponential utility provides a principled criterion for risk-sensitive sequential decision-making, but its nonlinear structure complicates reinforcement learning. A recent work \citep{thoppe2026reinforcement} addressed this difficulty by introducing a Bellman-compatible surrogate and two model-free fixed-point algorithms for optimizing it over stationary policies. However, their main convergence results are asymptotic. In this work, we establish finite-time rates of O~(1/n) for the aforementioned two algorithms under asynchronous Markovian sampling, where n is the iteration index and O~ hides logarithmic expressions. Importantly, we employ parameter-free choices for the stepsize parameter to derive these rate results. For the algorithmically simpler one-timescale method, the main challenge is that its update equation is not directly aligned with the contraction geometry of its underlying power-law operator. We overcome this mismatch by exploiting the boundedness, monotonicity, and homogeneity of the operator to obtain a local pseudo-contraction property for the relative-error dynamics. We then use a Moreau-envelope-based Lyapunov function and Polyak--Ruppert averaging to obtain the stated convergence rate with parameter-free stepsizes. For the two-timescale method, the main challenge is to control a tracking error on the faster timescale. These results provide the first finite-time guarantees for model-free discounted exponential-utility reinforcement learning.
Ankur Naskar, Vivek T A, Aditya Kumar +2
Indian Institute of Science, Bengaluru, India · Indian Institute of Technology Madras, Chennai, India
Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety-critical applications where online exploration is costly or unsafe. Diffusion-based decision-making methods have recently achieved strong performance in offline RL by modeling rich, multimodal trajectory distributions. However, existing diffusion planners are typically risk-neutral and therefore may overlook rare but catastrophic outcomes that are crucial in real-world deployment. In this work, we propose RS-Diffuser, a risk-sensitive offline diffusion planning framework that combines diffusion-based trajectory generation with distributional value critics. RS-Diffuser learns a diffusion planner over future state trajectories, a separate inverse dynamics model for action decoding, and a Monte Carlo distributional critic that estimates the full return distribution of candidate plans through quantile regression. At sampling time, we incorporate a risk-sensitive guidance signal into the denoising process, using gradients computed from tail-aware objectives such as Conditional Value at Risk to steer generation toward desired risk profiles. As a result, a single trained model can flexibly produce risk-averse, risk-neutral, or risk-seeking behaviors by changing only the inference-time risk parameter. Extensive experiments on risk-sensitive D4RL and risky robot navigation benchmarks demonstrate that RS-Diffuser achieves state-of-the-art performance, improving both overall return and worst-case robustness while reducing safety violations.
Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement learning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.
Manoosh Samiei, Doina Precup, Paul Masset
Department of Computer Science, McGill University, Canada · 2Mila - Quebec Artificial Intelligence Institute · Department of Psychology, McGill University, Canada