Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study
Organizations: Department of Industrial Engineering and Operations Research, Columbia University, New York, NY, 10027, USA. · Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, Hong Kong, N.T. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY, 10027, USA.
Abstract
We study continuous-time mean--variance portfolio selection in markets where stock prices are diffusion processes driven by observable factors that are also diffusion processes, yet the coefficients of these processes are unknown. Based on the recently developed reinforcement learning (RL) theory for diffusion processes, we present a general data-driven RL approach that learns the pre-committed investment strategy directly without attempting to learn or estimate the market coefficients. For multi-stock Black--Scholes markets without factors, we further devise an algorithm and prove its performance guarantee by deriving a sublinear regret bound in terms of the Sharpe ratio. We then carry out an extensive empirical study implementing this algorithm to compare its performance and trading characteristics, evaluated under a host of common metrics, with a large number of widely employed portfolio allocation strategies on S&P 500 constituents. The results demonstrate that the proposed continuous-time RL strategy is consistently among the best, especially in a volatile bear market, and decisively outperforms the model-based continuous-time counterparts by significant margins.
Figures & tables
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| S&P500 | 4.077% (0.000%) | 0.189 (0.000) | 0.216 (0.000) | 0.341 (0.000) | 0.072 (0.000) | 0.568 (0.000) | 1021 (0) |
| ew | 10.343% (0.185%) | 0.218 (0.002) | 0.483 (0.011) | 0.789 (0.018) | 0.182 (0.005) | 0.585 (0.008) | 544 (31) |
| mv | 4.226% (0.274%) | 0.148 (0.002) | 0.304 (0.020) | 0.491 (0.032) | 0.113 (0.008) | 0.443 (0.013) | 1349 (77) |
| js | 7.493% (0.537%) | 0.240 (0.019) | 0.397 (0.023) | 0.644 (0.038) | 0.156 (0.011) | 0.577 (0.016) | 1124 (71) |
| bl | 5.180% (1.962%) | 0.618 (0.092) | 0.195 (0.027) | 0.350 (0.041) | 0.079 (0.020) | 0.851 (0.013) | 1180 (70) |
| lw | 3.797% (0.226%) | 0.154 (0.002) | 0.261 (0.016) | 0.421 (0.026) | 0.097 (0.007) | 0.456 (0.013) | 1452 (80) |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| ew | 0% | 5.798% | 10.000% | 0.000% | 100.000% | 100.000% |
| mv | 0% | 16.437% | 113.971% | 61.028% | 113.558% | 260.610% |
| js | 1% | 15.345% | 171.726% | 106.333% | 140.192% | 542.149% |
| bl | 16% | 15.900% | 271.773% | 150.030% | 149.242% | 1000.542% |
| lw | 0% | 18.745% | 116.278% | 82.500% | 117.245% | 389.493% |
| ff | 0% | 8.841% | 116.479% | 53.619% | 96.796% | 274.978% |
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| S&P500 | 4.077% (0.000%) | 0.189 (0.000) | 0.216 (0.000) | 0.341 (0.000) | 0.072 (0.000) | 0.568 (0.000) | 1021 (0) |
| ew | 9.959% (0.185%) | 0.218 (0.002) | 0.465 (0.011) | 0.760 (0.018) | 0.174 (0.005) | 0.588 (0.008) | 568 (31) |
| mv | 3.203% (0.279%) | 0.148 (0.002) | 0.234 (0.019) | 0.381 (0.031) | 0.085 (0.007) | 0.462 (0.013) | 1490 (72) |
| js | 6.507% (0.548%) | 0.240 (0.019) | 0.352 (0.023) | 0.572 (0.037) | 0.135 (0.010) | 0.587 (0.016) | 1199 (64) |
| bl | 4.186% (1.958%) | 0.618 (0.092) | 0.172 (0.027) | 0.312 (0.041) | 0.066 (0.020) | 0.854 (0.013) | 1378 (88) |
| lw | 2.635% (0.230%) | 0.154 (0.002) | 0.185 (0.016) | 0.300 (0.026) | 0.067 (0.006) | 0.478 (0.013) | 2027 (120) |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| ew | 0% | 5.798% | 10.000% | 0.000% | 100.000% | 100.000% |
| mv | 0% | 16.432% | 113.971% | 61.028% | 113.558% | 260.610% |
| js | 1% | 15.341% | 171.726% | 106.333% | 140.192% | 542.149% |
| bl | 16% | 15.912% | 271.773% | 150.030% | 149.242% | 1000.542% |
| lw | 0% | 18.741% | 116.278% | 82.500% | 117.245% | 389.493% |
| ff | 0% | 8.840% | 116.479% | 53.619% | 96.796% | 274.978% |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Strategy | Symbol | Reference / Description |
| Buy-and-hold S&P 500 index | S&P 500 | Capitalization‑weighted U.S. market index |
| Equally weighted allocation | ew | DeMiguel et al., 2009b () |
| Mean–variance (single‑period) | mv | Markowitz, (1952) |
| Minimum variance | min_v | Markowitz, (1952) |
| James–Stein shrinkage (mean) | js | Jorion, (1986) |
| Ledoit–Wolf shrinkage (covariance) | lw | Ledoit and Wolf, (2003) |
| S&P500 | ew | mv | js | bl | lw | ff | rp | min_v | pmv | ctmv | ddpg | ppo | CTRL | |
| S&P500 | 1 | 1 | 1 | 1 | 0.953 | 1 | 1 | 1 | 0.997 | 0.143 | 0.826 | 0.999 | 1 | |
| ew | 2.01e-18 | 3.83e-16 | 8.1e-06 | 8.86e-16 | 4.01e-18 | 1.12e-07 | 0.274 | 0.578 | 1.34e-14 | 2.97e-16 | 6.71e-13 | 2.99e-12 | 0.999 | |
| mv | 0.000184 | 1 | 1 | 0.941 | 9.82e-05 | 1 | 1 | 1 | 0.248 | 0.000963 | 0.0824 | 0.561 | 1 | |
| js | 1.98e-12 | 1 | 1.79e-16 | 1.1e-06 | 8.77e-15 | 0.974 | 0.999 | 1 | 2.69e-09 | 2.28e-11 | 2.61e-06 | 0.000232 | 1 | |
| bl | 0.000381 | 1 | 0.0589 | 1 | 0.00166 | 1 | 1 | 1 | 0.0317 | 0.000191 | 0.0463 | 0.373 | 1 | |
| lw | 0.047 | 1 | 1 | 1 | 0.998 | 1 | 1 | 1 | 0.954 | 0.0527 | 0.462 | 0.958 | 1 |
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| S&P500 | 4.077% (0.000%) | 0.189 (0.000) | 0.216 (0.000) | 0.341 (0.000) | 0.072 (0.000) | 0.568 (0.000) | 1021 (0) |
| ew | 10.247% (0.185%) | 0.218 (0.002) | 0.478 (0.011) | 0.782 (0.018) | 0.180 (0.005) | 0.586 (0.008) | 551 (31) |
| mv | 3.970% (0.275%) | 0.148 (0.002) | 0.287 (0.020) | 0.463 (0.031) | 0.106 (0.008) | 0.447 (0.013) | 1359 (70) |
| js | 7.246% (0.540%) | 0.240 (0.019) | 0.386 (0.023) | 0.626 (0.037) | 0.151 (0.011) | 0.579 (0.016) | 1126 (65) |
| bl | 4.932% (1.960%) | 0.618 (0.092) | 0.189 (0.027) | 0.340 (0.041) | 0.076 (0.020) | 0.851 (0.013) | 1250 (79) |
| lw | 3.506% (0.227%) | 0.154 (0.002) | 0.242 (0.016) | 0.391 (0.026) | 0.090 (0.006) | 0.462 (0.013) | 1487 (78) |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| ew | 0% | 5.798% | 10.000% | 0.000% | 100.000% | 100.000% |
| mv | 0% | 16.435% | 113.971% | 61.028% | 113.558% | 260.610% |
| js | 1% | 15.344% | 171.726% | 106.333% | 140.192% | 542.149% |
| bl | 16% | 15.903% | 271.773% | 150.030% | 149.242% | 1000.542% |
| lw | 0% | 18.744% | 116.278% | 82.500% | 117.245% | 389.493% |
| ff | 0% | 8.840% | 116.479% | 53.619% | 96.796% | 274.978% |
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| S&P500 | -2.633% (0.000%) | 0.222 (0.000) | -0.118 (0.000) | -0.190 (0.000) | -0.046 (0.000) | 0.568 (0.000) | N/A (N/A) |
| ew | 7.467% (0.286%) | 0.252 (0.003) | 0.301 (0.012) | 0.497 (0.020) | 0.133 (0.006) | 0.585 (0.008) | 259 (3) |
| mv | -0.166% (0.358%) | 0.169 (0.004) | 0.013 (0.019) | 0.027 (0.030) | 0.011 (0.008) | 0.431 (0.012) | 1488 (17) |
| js | 0.242% (0.636%) | 0.282 (0.026) | 0.064 (0.020) | 0.105 (0.031) | 0.027 (0.010) | 0.570 (0.016) | 476 (5) |
| bl | -3.817% (1.830%) | 0.731 (0.123) | -0.030 (0.020) | -0.021 (0.032) | -0.030 (0.018) | 0.850 (0.014) | 223 (2) |
| lw | 0.411% (0.338%) | 0.178 (0.003) | 0.041 (0.019) | 0.071 (0.030) | 0.024 (0.008) | 0.440 (0.012) | 1790 (43) |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| ew | 0% | 6.559% | 10.000% | 0.000% | 100.000% | 100.000% |
| mv | 0% | 13.854% | 81.364% | 61.028% | 94.358% | 260.610% |
| js | 1% | 13.370% | 136.498% | 106.333% | 114.192% | 461.708% |
| bl | 16% | 18.404% | 271.773% | 150.030% | 130.610% | 1000.542% |
| lw | 0% | 16.854% | 100.599% | 82.500% | 103.348% | 389.493% |
| ff | 0% | 10.359% | 93.438% | 53.619% | 94.511% | 274.978% |
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| S&P500 | 11.070% (0.000%) | 0.148 (0.000) | 0.750 (0.000) | 1.173 (0.000) | 0.560 (0.000) | 0.198 (0.000) | 81 (0) |
| ew | 13.335% (0.248%) | 0.177 (0.002) | 0.772 (0.020) | 1.249 (0.032) | 0.526 (0.018) | 0.270 (0.006) | 195 (11) |
| mv | 8.873% (0.375%) | 0.123 (0.001) | 0.736 (0.031) | 1.224 (0.052) | 0.515 (0.031) | 0.207 (0.008) | 303 (25) |
| js | 15.379% (0.602%) | 0.177 (0.006) | 0.941 (0.031) | 1.564 (0.053) | 0.761 (0.034) | 0.240 (0.011) | 177 (17) |
| bl | 15.348% (2.260%) | 0.419 (0.049) | 0.647 (0.047) | 1.074 (0.071) | 0.477 (0.038) | 0.494 (0.025) | 242 (29) |
| lw | 7.359% (0.330%) | 0.123 (0.002) | 0.608 (0.027) | 1.004 (0.045) | 0.382 (0.023) | 0.225 (0.008) | 430 (37) |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| ew | 0% | 5.036% | 10.000% | 0.000% | 100.000% | 100.000% |
| mv | 0% | 19.019% | 113.971% | 58.252% | 132.757% | 260.003% |
| js | 1% | 17.320% | 171.726% | 82.796% | 166.192% | 542.149% |
| bl | 16% | 13.396% | 168.732% | 85.625% | 167.874% | 574.338% |
| lw | 0% | 20.636% | 116.278% | 62.322% | 131.142% | 295.426% |
| ff | 0% | 7.323% | 116.479% | 38.422% | 99.081% | 225.771% |
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| CTRL-baseline | 12.211% (0.219%) | 0.232 (0.003) | 0.531 (0.010) | 0.865 (0.017) | 0.200 (0.005) | 0.621 (0.009) | 521 (30) |
| CTRL-lr3/2 | 12.202% (0.220%) | 0.232 (0.003) | 0.530 (0.010) | 0.864 (0.017) | 0.200 (0.005) | 0.621 (0.009) | 523 (30) |
| CTRL-lr2/1 | 12.192% (0.221%) | 0.232 (0.003) | 0.530 (0.010) | 0.863 (0.017) | 0.200 (0.005) | 0.621 (0.009) | 529 (30) |
| CTRL-lr2/3 | 12.217% (0.219%) | 0.233 (0.003) | 0.531 (0.010) | 0.865 (0.017) | 0.201 (0.005) | 0.620 (0.009) | 519 (30) |
| CTRL-lr1/2 | 12.220% (0.219%) | 0.233 (0.003) | 0.531 (0.010) | 0.866 (0.017) | 0.201 (0.005) | 0.620 (0.009) | 513 (30) |
| CTRL-gamma3/2 | 12.211% (0.219%) | 0.232 (0.003) | 0.531 (0.010) | 0.865 (0.017) | 0.200 (0.005) | 0.621 (0.009) | 521 (30) |
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| CTRL-baseline | 12.115% (0.219%) | 0.232 (0.003) | 0.527 (0.010) | 0.858 (0.017) | 0.199 (0.005) | 0.621 (0.009) | 523 (30) |
| CTRL-lr3/2 | 12.106% (0.220%) | 0.232 (0.003) | 0.526 (0.010) | 0.857 (0.017) | 0.198 (0.005) | 0.622 (0.009) | 528 (30) |
| CTRL-lr2/1 | 12.097% (0.220%) | 0.232 (0.003) | 0.526 (0.010) | 0.856 (0.017) | 0.198 (0.005) | 0.622 (0.009) | 537 (30) |
| CTRL-lr2/3 | 12.121% (0.218%) | 0.233 (0.003) | 0.527 (0.010) | 0.859 (0.017) | 0.199 (0.005) | 0.621 (0.009) | 522 (30) |
| CTRL-lr1/2 | 12.124% (0.218%) | 0.233 (0.003) | 0.527 (0.010) | 0.859 (0.017) | 0.199 (0.005) | 0.621 (0.009) | 521 (30) |
| CTRL-gamma3/2 | 12.115% (0.219%) | 0.232 (0.003) | 0.527 (0.010) | 0.858 (0.017) | 0.199 (0.005) | 0.621 (0.009) | 523 (30) |
| Return | Volatility | Sharpe | Sortino | Calmar | MDD | RT | |
| CTRL-baseline | 11.828% (0.218%) | 0.233 (0.003) | 0.514 (0.010) | 0.837 (0.017) | 0.193 (0.005) | 0.623 (0.009) | 538 (31) |
| CTRL-lr3/2 | 11.820% (0.219%) | 0.232 (0.003) | 0.514 (0.010) | 0.837 (0.017) | 0.193 (0.005) | 0.624 (0.009) | 548 (31) |
| CTRL-lr2/1 | 11.812% (0.219%) | 0.232 (0.003) | 0.514 (0.010) | 0.836 (0.017) | 0.193 (0.005) | 0.624 (0.009) | 556 (31) |
| CTRL-lr2/3 | 11.834% (0.217%) | 0.233 (0.003) | 0.515 (0.010) | 0.838 (0.017) | 0.194 (0.005) | 0.623 (0.009) | 537 (31) |
| CTRL-lr1/2 | 11.836% (0.217%) | 0.233 (0.003) | 0.515 (0.010) | 0.838 (0.017) | 0.194 (0.005) | 0.623 (0.009) | 536 (31) |
| CTRL-gamma3/2 | 11.828% (0.218%) | 0.232 (0.003) | 0.514 (0.010) | 0.837 (0.017) | 0.193 (0.005) | 0.623 (0.009) | 538 (31) |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| CTRL-baseline | 0% | 5.679% | 18.778% | 1.903% | 102.301% | 124.034% |
| CTRL-lr3/2 | 0% | 5.658% | 18.903% | 1.930% | 102.479% | 125.026% |
| CTRL-lr2/1 | 0% | 5.637% | 18.901% | 1.957% | 102.657% | 126.016% |
| CTRL-lr2/3 | 0% | 5.693% | 18.653% | 1.885% | 102.179% | 123.366% |
| CTRL-lr1/2 | 0% | 5.700% | 18.590% | 1.876% | 102.118% | 122.977% |
| CTRL-gamma3/2 | 0% | 5.679% | 18.780% | 1.903% | 102.297% | 124.026% |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| CTRL-baseline | 0% | 5.679% | 18.778% | 1.903% | 102.301% | 124.034% |
| CTRL-lr3/2 | 0% | 5.658% | 18.903% | 1.930% | 102.479% | 125.026% |
| CTRL-lr2/1 | 0% | 5.638% | 18.901% | 1.957% | 102.657% | 126.016% |
| CTRL-lr2/3 | 0% | 5.693% | 18.653% | 1.885% | 102.179% | 123.366% |
| CTRL-lr1/2 | 0% | 5.701% | 18.590% | 1.876% | 102.118% | 122.977% |
| CTRL-gamma3/2 | 0% | 5.679% | 18.780% | 1.903% | 102.297% | 124.026% |
| BKR | TO | MaxL | MaxS | GE | GEMax | |
| CTRL-baseline | 0% | 5.680% | 18.778% | 1.903% | 102.301% | 124.034% |
| CTRL-lr3/2 | 0% | 5.659% | 18.903% | 1.930% | 102.479% | 125.026% |
| CTRL-lr2/1 | 0% | 5.638% | 18.901% | 1.957% | 102.657% | 126.016% |
| CTRL-lr2/3 | 0% | 5.694% | 18.653% | 1.885% | 102.179% | 123.366% |
| CTRL-lr1/2 | 0% | 5.701% | 18.590% | 1.876% | 102.118% | 122.977% |
| CTRL-gamma3/2 | 0% | 5.680% | 18.780% | 1.903% | 102.297% | 124.026% |
Portfolio Optimization
Portfolio optimization selects asset weights to balance expected return, risk, diversification, and trading constraints.
See all 71 papers