Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study
Authors: Yilie Huang, Yanwei Jia, Xun Yu Zhou
Organizations: Department of Industrial Engineering and Operations Research, Columbia University, New York, NY, 10027, USA. · Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, Hong Kong, N.T. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY, 10027, USA.
We study continuous-time mean--variance portfolio selection in markets where stock prices are diffusion processes driven by observable factors that are also diffusion processes, yet the coefficients of these processes are unknown. Based on the recently developed reinforcement learning (RL) theory for diffusion processes, we present a general data-driven RL approach that learns the pre-committed investment strategy directly without attempting to learn or estimate the market coefficients. For multi-stock Black--Scholes markets without factors, we further devise an algorithm and prove its performance guarantee by deriving a sublinear regret bound in terms of the Sharpe ratio. We then carry out an extensive empirical study implementing this algorithm to compare its performance and trading characteristics, evaluated under a host of common metrics, with a large number of widely employed portfolio allocation strategies on S&P 500 constituents. The results demonstrate that the proposed continuous-time RL strategy is consistently among the best, especially in a volatile bear market, and decisively outperforms the model-based continuous-time counterparts by significant margins.
Figures & tables
Figure 1: Wealth trajectories under the proposed CTRL algorithm and 13 alternative methods over 100 independent experiments, each with 10 randomly selected stocks (except the S&P 500 index), from 2000 to 2020. The solid line shows the median across experiments at each time, and the shaded region is the interquartile range.
Return
Volatility
Sharpe
Sortino
Calmar
MDD
RT
S&P500
4.077% (0.000%)
0.189 (0.000)
0.216 (0.000)
0.341 (0.000)
0.072 (0.000)
0.568 (0.000)
1021 (0)
ew
10.343% (0.185%)
0.218 (0.002)
0.483 (0.011)
0.789 (0.018)
0.182 (0.005)
0.585 (0.008)
544 (31)
mv
4.226% (0.274%)
0.148 (0.002)
0.304 (0.020)
0.491 (0.032)
0.113 (0.008)
0.443 (0.013)
1349 (77)
js
7.493% (0.537%)
0.240 (0.019)
0.397 (0.023)
0.644 (0.038)
0.156 (0.011)
0.577 (0.016)
1124 (71)
bl
5.180% (1.962%)
0.618 (0.092)
0.195 (0.027)
0.350 (0.041)
0.079 (0.020)
0.851 (0.013)
1180 (70)
lw
3.797% (0.226%)
0.154 (0.002)
0.261 (0.016)
0.421 (0.026)
0.097 (0.007)
0.456 (0.013)
1452 (80)
Table 1: Comparison of out-of-sample performance of different allocation methods from 2000 to 2020. We report annualized return, volatility, Sharpe ratio, Sortino ratio, and Calmar ratio, together with maximum drawdown (MDD) and recovery time (RT) in days, over 100 independent experiments. Each experiment is based on 10 randomly selected stocks (except for S&P 500 index). For each cell, the upper number is the average (over the 100 experiments) while the lower one with parentheses is the standard error.
BKR
TO
MaxL
MaxS
GE
GEMax
ew
0%
5.798%
10.000%
0.000%
100.000%
100.000%
mv
0%
16.437%
113.971%
61.028%
113.558%
260.610%
js
1%
15.345%
171.726%
106.333%
140.192%
542.149%
bl
16%
15.900%
271.773%
150.030%
149.242%
1000.542%
lw
0%
18.745%
116.278%
82.500%
117.245%
389.493%
ff
0%
8.841%
116.479%
53.619%
96.796%
274.978%
Table 2: Trading and exposure statistics over the period 2000–2020. BKR is the bankruptcy rate across experiments, TO is the average monthly turnover, MaxL and MaxS are the maximum single-asset long and short positions, respectively, GE is the average gross exposure, and GEMax is the maximum gross exposure observed across all experiments.
Return
Volatility
Sharpe
Sortino
Calmar
MDD
RT
S&P500
4.077% (0.000%)
0.189 (0.000)
0.216 (0.000)
0.341 (0.000)
0.072 (0.000)
0.568 (0.000)
1021 (0)
ew
9.959% (0.185%)
0.218 (0.002)
0.465 (0.011)
0.760 (0.018)
0.174 (0.005)
0.588 (0.008)
568 (31)
mv
3.203% (0.279%)
0.148 (0.002)
0.234 (0.019)
0.381 (0.031)
0.085 (0.007)
0.462 (0.013)
1490 (72)
js
6.507% (0.548%)
0.240 (0.019)
0.352 (0.023)
0.572 (0.037)
0.135 (0.010)
0.587 (0.016)
1199 (64)
bl
4.186% (1.958%)
0.618 (0.092)
0.172 (0.027)
0.312 (0.041)
0.066 (0.020)
0.854 (0.013)
1378 (88)
lw
2.635% (0.230%)
0.154 (0.002)
0.185 (0.016)
0.300 (0.026)
0.067 (0.006)
0.478 (0.013)
2027 (120)
Table 3: Net-of-cost performance over the full sample period (2000–2020) under 50 bps proportional transaction costs. We report annualized return, volatility, Sharpe ratio, Sortino ratio, and Calmar ratio, together with maximum drawdown (MDD) and recovery time (RT) in days, over 100 independent experiments. Each experiment is based on 10 randomly selected stocks (except for S&P 500 index). Numbers in parentheses are the standard errors across 100 experiments.
BKR
TO
MaxL
MaxS
GE
GEMax
ew
0%
5.798%
10.000%
0.000%
100.000%
100.000%
mv
0%
16.432%
113.971%
61.028%
113.558%
260.610%
js
1%
15.341%
171.726%
106.333%
140.192%
542.149%
bl
16%
15.912%
271.773%
150.030%
149.242%
1000.542%
lw
0%
18.741%
116.278%
82.500%
117.245%
389.493%
ff
0%
8.840%
116.479%
53.619%
96.796%
274.978%
Table 4: Trading and exposure characteristics under 50 bps proportional transaction costs over the full sample period (2000–2020). BKR denotes the bankruptcy rate across experiments. TO is the average monthly turnover. MaxL and MaxS report the maximum single-asset long and short positions, respectively. GE is the average gross exposure, and GEMax is the maximum gross exposure observed across all experiments.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Strategy
Symbol
Reference / Description
Buy-and-hold S&P 500 index
S&P 500
Capitalization‑weighted U.S. market index
Equally weighted allocation
ew
DeMiguel et al., 2009b ()
Mean–variance (single‑period)
mv
Markowitz, (1952)
Minimum variance
min_v
Markowitz, (1952)
James–Stein shrinkage (mean)
js
Jorion, (1986)
Ledoit–Wolf shrinkage (covariance)
lw
Ledoit and Wolf, (2003)
Appendix
Table 5: Alternative asset allocation strategies compared in this study.
Figure 2: Error of parameter ϕ1 The solid curves and the upper and lower boundaries of the shaded regions represent the average, 2.5% and 97.5% percentile of the error over 1000 independent simulation runs, respectively. The slope for ϕ1 by least squares regression is -1.09. The vertical and horizontal axes are on natural log-scale.
Figure 3: Error of parameter ϕ2 The solid curves and the upper and lower boundaries of the shaded regions represent the average, 2.5% and 97.5% percentile of the error over 1000 independent simulation runs, respectively. The slope for ϕ2 by least squares regression is -0.91. The vertical and horizontal axes are on natural log-scale.
Figure 4: Error of parameter w The solid curves and the upper and lower boundaries of the shaded regions represent the average, 2.5% and 97.5% percentile of the error over 1000 independent simulation runs, respectively. The slope for w by least squares regression is -0.97. The vertical and horizontal axes are on natural log-scale.
Figure 5: Cumulative regret rate in number of episodes. The solid blue curve and the upper and lower boundary of the shaded region represent the mean, 2.5% and 97.5% percentile of the regret over 1000 independent simulation runs, respectively. The red dashed line is the fitted value by linearly regressing the log average regret against the logarithm of the number of episodes starting from the 100th episode. The fitted slope by least squares regression is 0.456 . The vertical and horizontal axes are on natural log-scale.
Figure 6: Log–log plot of ∣Var(Z1)∣ versus ∣ϕ2−1∣ , showing how the exploration level ϕ2 affects the stability of the update for ϕ1 . The shaded area represents the 95% confidence interval.
Figure 7: Median wealth trajectories with interquartile ranges for CTRL, mv, ctmv, and ew over the full sample period.
Figure 8: Box plots of Sharpe ratios over the full sample period across 100 experiments. Extremely negative values are truncated at −1 .
Table 7: Net-of-cost performance over the full sample period (2000–2020) under 12.5 bps proportional transaction costs. We report annualized return, volatility, Sharpe ratio, Sortino ratio, and Calmar ratio, together with maximum drawdown (MDD) and recovery time (RT) in days, over 100 independent experiments. Each experiment is based on 10 randomly selected stocks (except for S&P 500 index). Numbers in parentheses are the standard errors across 100 experiments.
BKR
TO
MaxL
MaxS
GE
GEMax
ew
0%
5.798%
10.000%
0.000%
100.000%
100.000%
mv
0%
16.435%
113.971%
61.028%
113.558%
260.610%
js
1%
15.344%
171.726%
106.333%
140.192%
542.149%
bl
16%
15.903%
271.773%
150.030%
149.242%
1000.542%
lw
0%
18.744%
116.278%
82.500%
117.245%
389.493%
ff
0%
8.840%
116.479%
53.619%
96.796%
274.978%
Appendix
Table 8: Trading and exposure characteristics under 12.5 bps proportional transaction costs over the full sample period (2000–2020). BKR denotes the bankruptcy rate across experiments. TO is the average monthly turnover. MaxL and MaxS report the maximum single-asset long and short positions, respectively. GE is the average gross exposure, and GEMax is the maximum gross exposure observed across all experiments.
Return
Volatility
Sharpe
Sortino
Calmar
MDD
RT
S&P500
-2.633% (0.000%)
0.222 (0.000)
-0.118 (0.000)
-0.190 (0.000)
-0.046 (0.000)
0.568 (0.000)
N/A (N/A)
ew
7.467% (0.286%)
0.252 (0.003)
0.301 (0.012)
0.497 (0.020)
0.133 (0.006)
0.585 (0.008)
259 (3)
mv
-0.166% (0.358%)
0.169 (0.004)
0.013 (0.019)
0.027 (0.030)
0.011 (0.008)
0.431 (0.012)
1488 (17)
js
0.242% (0.636%)
0.282 (0.026)
0.064 (0.020)
0.105 (0.031)
0.027 (0.010)
0.570 (0.016)
476 (5)
bl
-3.817% (1.830%)
0.731 (0.123)
-0.030 (0.020)
-0.021 (0.032)
-0.030 (0.018)
0.850 (0.014)
223 (2)
lw
0.411% (0.338%)
0.178 (0.003)
0.041 (0.019)
0.071 (0.030)
0.024 (0.008)
0.440 (0.012)
1790 (43)
Appendix
Table 9: Comparison of out-of-sample performance of different allocation methods in the bear period (2000-2010) We report annualized return, volatility, Sharpe ratio, Sortino ratio, and Calmar ratio, together with maximum drawdown (MDD) and recovery time (RT) in days, over 100 independent experiments. Each experiment is based on 10 randomly selected stocks (except for S&P 500 index). For each cell, the upper number is the average (over the 100 experiments) while the lower one with parentheses is the standard error.
BKR
TO
MaxL
MaxS
GE
GEMax
ew
0%
6.559%
10.000%
0.000%
100.000%
100.000%
mv
0%
13.854%
81.364%
61.028%
94.358%
260.610%
js
1%
13.370%
136.498%
106.333%
114.192%
461.708%
bl
16%
18.404%
271.773%
150.030%
130.610%
1000.542%
lw
0%
16.854%
100.599%
82.500%
103.348%
389.493%
ff
0%
10.359%
93.438%
53.619%
94.511%
274.978%
Appendix
Table 10: Trading and exposure statistics over the bear period 2000–2010. BKR is the bankruptcy rate across experiments, TO is the average monthly turnover, MaxL and MaxS are the maximum single-asset long and short positions, respectively, GE is the average gross exposure, and GEMax is the maximum gross exposure observed across all experiments.
Return
Volatility
Sharpe
Sortino
Calmar
MDD
RT
S&P500
11.070% (0.000%)
0.148 (0.000)
0.750 (0.000)
1.173 (0.000)
0.560 (0.000)
0.198 (0.000)
81 (0)
ew
13.335% (0.248%)
0.177 (0.002)
0.772 (0.020)
1.249 (0.032)
0.526 (0.018)
0.270 (0.006)
195 (11)
mv
8.873% (0.375%)
0.123 (0.001)
0.736 (0.031)
1.224 (0.052)
0.515 (0.031)
0.207 (0.008)
303 (25)
js
15.379% (0.602%)
0.177 (0.006)
0.941 (0.031)
1.564 (0.053)
0.761 (0.034)
0.240 (0.011)
177 (17)
bl
15.348% (2.260%)
0.419 (0.049)
0.647 (0.047)
1.074 (0.071)
0.477 (0.038)
0.494 (0.025)
242 (29)
lw
7.359% (0.330%)
0.123 (0.002)
0.608 (0.027)
1.004 (0.045)
0.382 (0.023)
0.225 (0.008)
430 (37)
Appendix
Table 11: Comparison of out-of-sample performance of different allocation methods in the bull period (2010-2020). We report annualized return, volatility, Sharpe ratio, Sortino ratio, and Calmar ratio, together with maximum drawdown (MDD) and recovery time (RT) in days, over 100 independent experiments. Each experiment is based on 10 randomly selected stocks (except for S&P 500 index). For each cell, the upper number is the average (over the 100 experiments) while the lower one with parentheses is the standard error.
BKR
TO
MaxL
MaxS
GE
GEMax
ew
0%
5.036%
10.000%
0.000%
100.000%
100.000%
mv
0%
19.019%
113.971%
58.252%
132.757%
260.003%
js
1%
17.320%
171.726%
82.796%
166.192%
542.149%
bl
16%
13.396%
168.732%
85.625%
167.874%
574.338%
lw
0%
20.636%
116.278%
62.322%
131.142%
295.426%
ff
0%
7.323%
116.479%
38.422%
99.081%
225.771%
Appendix
Table 12: Trading and exposure statistics over the bull period 2010–2020. BKR is the bankruptcy rate across experiments, TO is the average monthly turnover, MaxL and MaxS are the maximum single-asset long and short positions, respectively, GE is the average gross exposure, and GEMax is the maximum gross exposure observed across all experiments.
Return
Volatility
Sharpe
Sortino
Calmar
MDD
RT
CTRL-baseline
12.211% (0.219%)
0.232 (0.003)
0.531 (0.010)
0.865 (0.017)
0.200 (0.005)
0.621 (0.009)
521 (30)
CTRL-lr3/2
12.202% (0.220%)
0.232 (0.003)
0.530 (0.010)
0.864 (0.017)
0.200 (0.005)
0.621 (0.009)
523 (30)
CTRL-lr2/1
12.192% (0.221%)
0.232 (0.003)
0.530 (0.010)
0.863 (0.017)
0.200 (0.005)
0.621 (0.009)
529 (30)
CTRL-lr2/3
12.217% (0.219%)
0.233 (0.003)
0.531 (0.010)
0.865 (0.017)
0.201 (0.005)
0.620 (0.009)
519 (30)
CTRL-lr1/2
12.220% (0.219%)
0.233 (0.003)
0.531 (0.010)
0.866 (0.017)
0.201 (0.005)
0.620 (0.009)
513 (30)
CTRL-gamma3/2
12.211% (0.219%)
0.232 (0.003)
0.531 (0.010)
0.865 (0.017)
0.200 (0.005)
0.621 (0.009)
521 (30)
Appendix
Table 13: Performance summary: No transaction cost.
This study develops and evaluates a deep reinforcement learning framework for dynamic portfolio allocation across global equity markets. The Soft Actor-Critic algorithm is used to learn continuous portfolio weights within a Markov Decision Process, incorporating transaction costs, turnover penalties, and diversification constraints into the reward function. Five model configurations are compared, varying in reward formulation, policy structure (flat versus hierarchical Dirichlet), portfolio constraints, and temporal encoder (LSTM versus Transformer), and evaluated via walk-forward optimization across sixteen out-of-sample folds spanning 2003-2026 on the Nasdaq-100, Nikkei 225, and Euro Stoxx 50. Results show that RL strategies achieve competitive risk-adjusted performance primarily in the Euro Stoxx 50, where statistically significant abnormal returns are observed, but the central hypothesis is only partially confirmed: no strategy achieves statistically significant excess returns relative to Buy and Hold under HAC-robust inference across all markets. Regime analysis reveals that RL adds the most value during periods of elevated uncertainty, while ensemble aggregation across markets improves risk-adjusted performance and confirms the benefits of geographic diversification.
Kamil Kashif, Robert Ślepaczuk
Quantitative Finance Research Group Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland · Quantitative Finance Research Group Department of Quantitative Finance and Machine Learning Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland
Deep reinforcement learning (DRL) frameworks for portfolio optimization have shown promise for their ability to learn allocation rules dynamically from market data. However, these models fail to account for fat-tailed returns, which characterize actual market behavior with more frequent extreme events. Furthermore, historical data is treated homogeneously, without accounting for temporal importance, leading models to fail during regime changes. We propose a new BAVAR-BLED algorithm that combines methods derived from Bayesian-Averaging Vector Autoregressive (BAVAR) and the Black-Litterman model using Elliptical Distributions (BLED) within a TD3 architecture. BAVAR captures a set of vector autoregressive representations that consider multi-scale temporal features, enabling adaptive allocation decisions based on regime-aware estimates of return expectations and dispersion matrices. These estimates serve as prior inputs to BLED, a model that uses Student's t-distributions, allowing for more realistic fat tail return estimates. The BAVAR-BLED algorithm uses transformer networks for view construction and CNNs for risk-aversion estimates, which modify dynamic allocation decisions based on market conditions. An evaluation of 29 Dow Jones Industrial Average constituents over a decade-long market period shows that BAVAR-BLED significantly outperforms state-of-the-art methods, achieving Sharpe and Sortino ratios of 1.72 and 2.70, respectively, and total returns of 57.26%.
We study reinforcement learning for controlled diffusion processes with unbounded continuous state spaces, bounded continuous actions, and polynomially growing rewards: settings that arise naturally in finance, economics, and operations research. To overcome the challenges of continuous and high-dimensional domains, we introduce a model-based algorithm that adaptively partitions the joint state-action space. The algorithm maintains estimators of drift, volatility, and rewards within each partition, refining the discretization whenever estimation bias exceeds statistical confidence. This adaptive scheme balances exploration and approximation, enabling efficient learning in unbounded domains. Our analysis establishes regret bounds that depend on the problem horizon, state dimension, reward growth order, and a newly defined notion of zooming dimension tailored to unbounded diffusion processes. The bounds recover existing results for bounded settings as a special case, while extending theoretical guarantees to a broader class of diffusion-type problems. Finally, we validate the effectiveness of our approach through numerical experiments, including applications to high-dimensional problems such as multi-asset mean-variance portfolio selection.
Hanqing Jin, Renyuan Xu, Yanzhao Yang
Mathematical Institute, University of Oxford · Department of Management Science and Engineering, Stanford University