Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting
Authors: Mehrdad Moghimi, Anthony Coache, Hyejin Ku
Organizations: Dept. of Mathematics and Statistics, York University, Toronto, Canada · Dept. of Mathematics, Imperial College London, London, United Kingdom
Distributional reinforcement learning (RL) is a powerful framework increasingly adopted in safety-critical domains for its ability to optimize risk-sensitive objectives. However, the role of the discount factor is often overlooked, as it is typically treated as a fixed parameter of the Markov decision process or tunable hyperparameter, with little consideration of its effect on the learned policy. In the literature, it is well-known that the discounting function plays a major role in characterizing time preferences of an agent, which an exponential discount factor cannot fully capture. Building on this insight, we propose a novel framework that supports flexible discounting of future rewards and optimization of risk measures in distributional RL. We provide a technical analysis of the optimality of our algorithms, show that our multi-horizon extension fixes issues raised with existing methodologies, and validate the robustness of our methods through extensive experiments. Our results highlight that discounting is a cornerstone in decision-making problems for capturing more expressive temporal and risk preferences profiles, with potential implications for real-world safety-critical applications.
Figures & tables
Figure 1: Impact of Time-Consistency. Our time-consistent method (blue) outperforms the time-inconsistent approach of Fedus et al. (2019) (red) by modeling the non-stationary policy.
Criterion
Discount
T1
T2
T3
T4
T5
T6
Optimal
E
γ=0.8
16.000
17.920
8.192
4.194
6.040
3.006
T2
E
k=1
10.000
8.750
4.167
2.500
4.500
2.917
T1
Entr0.5
γ=0.8
1.386
1.386
4.648
4.194
4.002
2.961
T3
Entr0.5
k=1
1.386
1.386
2.895
2.500
3.229
2.874
T5
Entr1.5
γ=0.8
0.462
0.462
3.739
4.194
3.146
2.877
T4
Entr1.5
k=1
0.462
0.462
2.128
2.500
2.462
2.794
T6
Table 1: Comparison of optimal plans under different criteria and discount functions.
π^t∗={π^t∗πRN∗if t<T′if t≥T′
Algorithm 1 Distributional VI for Infinite-Horizon MDPs with General Discounting
Figure 2: Preference reversals in GBWM. Monte-Carlo probabilities of achieving the two goals for a risk-neutral (left) and risk-sensitive (right) agent, with various discounting (colors) and environment settings (shapes). Standard errors, not included for clarity, are provided in Appendix H.3 .
Figure 3: Relative performance of our Time-Consistent algorithm on 50 Atari games. Bars show percent improvement over the time-inconsistent baseline (3 seeds). Our method outperforms in 39/50 games, with 39.89% mean and 18.14% median gains, highlighting the benefit of time-consistency under hyperbolic discounting.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: From interest rates to discount functions. Comparison of exponential with γ=0.99 (Orange), hyperbolic with k=0.01 (Green), and CIR-based discount factors with parameters a=0.05 , b=0.01 , σ=0.1 , and r0=0.01 (Blue)
Figure 5: Utility functions for common OCE risk measures. Detailed formulations are provided in Table 2 .
Risk Name
Parameter
Formula
Utility function f(x)
Mean
None
E[G]
x
CVaR
τ∈(0,1]
CVaRτ(G)
τ−1(x)−
Mean-CVaR
κ1∈[0,1]
κ1E[G]+(1−κ1)⋅CVaRτ(G)
κ1(x)++κ2(x)−, where
τ∈(0,1]
κ2=τ−1(1−κ1)+κ1
Entropic Risk
β∈(0,∞)
Entrβ(G)
β1(1−exp(−βx))
Mean-Variance
κ∈(0,∞)
E[G]−κ⋅Var(G)
(x−κx2)I{x≤2κ1}
Appendix
Table 2: Common Risk Measures as OCEs.
Portfolio weights
Portfolio number
U.S. Bonds
Int’l Stocks
U.S. Stocks
1
0.9098
0.0225
0.0677
2
0.8500
0.0033
0.1467
3
0.7903
−0.0160
0.2257
4
0.7305
−0.0352
0.3047
5
0.6707
−0.0545
0.3837
Appendix
Table 3: Portfolio weights, where the most conservative (portfolio 1) has an expected return of 0.0526 and volatility of 0.0374 , while the most aggressive (portfolio 15) has an expectation of 0.0886 and standard deviation of 0.1954 .
Method
E[U]
P[yT/2≥100]
P[ψT/2=1]
P[ψT=1]
uT/2(1)=1000 , uT(1)=1000 , T=10
E , γ=1.00
1104(24.54)
0.848(0.017)
0.848(0.017)
0.256(0.008)
E , γ=0.99
1112(11.81)
0.845(0.040)
0.845(0.040)
0.267(0.029)
E , k=0.05
1108(17.49)
0.864(0.020)
0.864(0.020)
0.244(0.037)
uT/2(1)=1000 , uT(1)=2000 , T=10
E , γ=1.00
1782(10.07)
0.855(0.016)
0.237(0.027)
0.772(0.012)
Appendix
Table 4: Evaluation of the policies learned by Algorithm 2 for a risk-neutral agent with cT/2(1)=100 and cT(1)=150 over 10,000 episodes, evaluated for 3 different runs. Standard errors are reported in round brackets.
Method
E[U]
P[yT/2≥100]
P[ψT/2=1]
P[ψT=1]
uT/2(1)=1000 , uT(1)=1000 , T=10
CVaR0.1 , γ=1.00
999(0.036)
0.998(0.0005)
0.998(0.0005)
0.001(0.0004)
CVaR0.1 , k=0.05
999(1.05)
0.994(0.0036)
0.994(0.0036)
0.005(0.0025)
uT/2(1)=1000 , uT(1)=2000 , T=10
CVaR0.1 , γ=1.00
1007(4.15)
0.986(0.0084)
0.986(0.0084)
0.010(0.0061)
CVaR0.1 , k=0.05
1020(18.05)
0.977(0.0202)
0.977(0.0202)
0.022(0.0148)
Appendix
Table 5: Evaluation of the policies learned by Algorithm 2 for a risk-sensitive agent with cT/2(1)=100 and cT(1)=150 over 10,000 episodes, evaluated for 3 different runs. Standard errors are reported in round brackets.
Hyperparameter
Value
Optimizer
Adam
Learning rate
5×10−5
Batch size B
32
Replay buffer size
1,000,000 transitions
Learning starts
50,000 steps
Training frequency
Every 4 environment steps
Appendix
Table 6: Atari hyperparameters used for the multi-horizon QR-DQN experiments.
Figure 6: Relative performance improvement of our Time-Consistent algorithm across 50 Atari games. Each bar shows the percentage improvement in mean return for the Time-Consistent policy over the Time-aware Time-Inconsistent baseline, averaged over 3 seeds per game. The Time-Consistent policy outperforms the baseline in 33 out of 50 games. Across all games, it achieves a mean improvement of 15.73% and a median improvement of 8.02%, demonstrating the benefits of maintaining time-consistency under hyperbolic discounting.
Figure 7: Comparison on Atari 2600. Learning curves over 10 million steps (40 million frames) for 50 Atari games. Our Time-Consistent agent (Blue) is compared against the Time-Inconsistent hyperbolic baseline from Fedus et al. (2019) (Red) and a time-aware variant of this baseline (Green). Shaded regions represent 95% bootstrap confidence intervals across 3 seeds. The results demonstrate that properly handling the non-stationarity of the optimal policy is critical for effective learning with general discount functions. Note that the reported mean returns are computed using the clipped rewards commonly used during training for stability, rather than the unclipped episodic returns.
Figure 8: Relative performance improvement of our Time-Consistent algorithm over exponentially discounted QR-DQN across 50 Atari games. Each bar shows the percentage improvement in mean return for the Time-Consistent policy over the Exponential baseline with γ=0.99 , averaged over 3 seeds per game. The Time-Consistent policy outperforms the baseline in 40 out of 50 games, with a median improvement of 51.3%.
Figure 9: Relative performance improvement of our Time-Consistent algorithm over the Multi-Horizon (largest γ ) agent across Atari games. Each bar shows the percentage improvement in mean return, averaged over 3 seeds per game. The Time-Consistent policy outperforms the baseline in 36 out of 50 games, with a median improvement of 17.0% over the 49 games shown. Enduro is not shown because the percentage is undefined when the baseline return is zero; we count it among the 36 games since the Time-Consistent policy obtains a positive return.
Figure 10: Relative performance improvement of the Multi-Horizon (largest γ ) agent over exponentially discounted QR-DQN across 50 Atari games. Each bar shows the percentage improvement in mean return, averaged over 3 seeds per game. The Multi-Horizon agent outperforms the Exponential baseline in 30 out of 50 games, with a median improvement of 9.6%.
Figure 11: Comparison with exponentially discounted baselines on Atari 2600. Learning curves for 50 Atari games. Our Time-Consistent agent (Blue) is compared against the Multi-Horizon (largest γ ) agent (Orange) and exponentially discounted QR-DQN with γ=0.99 (Purple). Shaded regions represent 95% bootstrap confidence intervals across 3 seeds. Mean returns are computed using the clipped rewards used during training.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Jul 22, 2026·Manoosh Samiei, Doina Precup, Paul MassetContinual RL
Department of Computer Science, McGill University, Canada · 2Mila - Quebec Artificial Intelligence Institute · Department of Psychology, McGill University, Canada