This paper proposes a machine learning assisted portfolio optimization framework designed for low data environments and regime uncertainty. We construct a teacher student learning pipeline in which a Conditional Value at Risk (CVaR) optimizer generates supervisory labels, and neural models (Bayesian and deterministic) are trained using both real and synthetically augmented data. The synthetic data is generated using a factor based model with t copula residuals, enabling training beyond the limited real sample of 104 labeled observations. We evaluate four student models under a structured experimental framework comprising (i) controlled synthetic experiments (3 x 5 seed grid), (ii) in-distribution real market evaluation (C2A) and (iii) cross-universe generalization (D2A). In real-market settings, models are deployed using a rolling evaluation protocol where a frozen pretrained model is periodically fine tuned on recent observations and reset to its base state, ensuring stability while allowing limited adaptation. Results show that student models can match or outperform the CVaR teacher in several settings, while achieving improved robustness under regime shifts and reduced turnover. These findings suggest that hybrid optimization learning approaches can enhance portfolio construction in data constrained environments
Figures & tables
Figure 1: Distribution of annualized Sharpe ratios across 15 runs per model (GRID_3 × 5, synthetic test set). Boxes show interquartile range, horizontal lines indicate medians, whiskers extend to 1.5 × IQR.
Figure 2: Sharpe–CVaR efficiency frontier across all 120 runs (8 models × 15 runs, GRID_3 × 5). BNN-S (orange) clusters in the Pareto optimal region (top-left), achieving high Sharpe with contained tail risk. Mean-Variance (blue) delivers equivalent Sharpe but with CVaR 3× worse, rendering it Pareto-dominated for risk-averse applications.
Figure 3: Mean weekly turnover vs annualized Sharpe across all runs (GRID_3 × 5). Yellow band (10–15% turnover) marks the optimal zone. BNN-S (orange) self-regulates into this band; DNN-S (green) overtrades at 20–26%, frequently degrading to Sharpe 0.9–1.2 despite high potential. DNN-sup is an exception among deterministic models, with low turnover ( ≈8.6% ) rather than overtrading. Mean-Variance and Risk-Parity under-trade ( <2% ), missing time-varying opportunities.
Figure 4: Pairwise win-rate matrix: P(row model > column model) across 15 GRID_3 × 5 runs. BNN-S achieves 100% win-rate against Risk-Parity and 80% against the Teacher, but only 27% against Min-Var. Distilled models (BNN-S, DNN-S) systematically outperform supervised variants (BNN-sup, DNN-sup).
Figure 5: Sharpe ratio vs row-wise correlation with teacher allocations across all 120 runs. R2<0.1 suggests that literal imitation fidelity does not predict out-of-sample performance. BNN-S runs (orange) with 85–90% teacher similarity span Sharpe 0.6–2.0, while Mean-Variance (blue) achieves 1.4–1.9 Sharpe with <10% similarity.
Figure 6: Sharpe distributions under seed 42 (volatility stress regime, 2015–2016 spike and 2018 Q4 correction). All models degrade relative to aggregate performance: Mean-Variance 1.684 ( −12% ), BNN-S 0.980 ( −46% ), Risk-Parity 0.690 ( −45% ). BNN-S retains a relative advantage over Risk-Parity despite severe degradation.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Synthetic data validation. Top: Per-asset annualised volatility distribution, real (blue) vs. synthetic (orange). Distributions overlap substantially in the 15–25% range; synthetic returns exhibit a slightly heavier right tail consistent with t -copula ( ν=6 ) calibration. Bottom: Element-wise difference between real and synthetic correlation matrices (Real − Synthetic). Most entries fall within ±0.15 ; structured deviations in USO and Indian equity names reflect limitations of static factor loadings for regime-dependent assets.
Figure 8: Sharpe distributions under seed 32 (bull market regime, 2016–2018 tech expansion). DNN-S achieves peak performance (Sharpe 2.15), outpacing Mean-Variance (1.99) and BNN-S (1.83). Momentum strategies thrive in persistent trending environments, giving deterministic overweighting of high-momentum assets a structural advantage. Risk-Parity trails at 1.30.
Figure 9: Sharpe distributions under seed 52 (mixed regime). Mean-Variance recovers partially (1.38) relative to seed 42, and DNN-S leads among ML models (1.22). Risk-Parity exhibits catastrophic failure (0.45, −66% relative to seed 32), confirming that equal-risk-weighting is particularly fragile under rapid regime transitions.
Figure 10: Cumulative wealth over ∼ 350 weeks (seed 32). Mean-Variance (green) exhibits exponential growth after week 200 ( 9× terminal wealth), capturing bull market momentum through concentrated positions. ML models (BNN-S, DNN-S) maintain steady 2.3× growth with substantially lower drawdowns. The divergence at week 200 is consistent with a regime shift from balanced to trending markets.
Figure 11: Distribution of CVaR (95%) expressed as weekly loss percentage across 15 runs per model (GRID_3 × 5). Lower values indicate worse tail risk. Mean-Variance exhibits median CVaR of −4.5% with wide dispersion, approximately 2.4× worse than BNN-S ( −1.8% ). BNN models show tighter distributions ( ±0.3% ) than DNN variants ( ±1.0% ), consistent with epistemic uncertainty reducing overconfident extreme positions [ Blundell et al., 2015 ] .
Figure 12: Mean Sharpe ratios with 95% bootstrap confidence intervals ( n=1000 resamples). Bootstrap confidence intervals show BNN-S with higher mean Sharpe than the Teacher, though intervals show partial overlap given the small sample ( n=15 ). Mean-Variance confidence interval ( [1.56,1.82] ) overlaps with BNN-S ( [0.98,1.45] ), and its tail-risk disqualification (Figure 11 ) renders raw Sharpe superiority misleading for risk-averse deployment.
Figure 13: Stacked win-rate heatmap across all model pairs (GRID_3 × 5, 15 runs). Each row represents one model, columns represent opponents, and color intensity indicates win percentage. Model 0 corresponds to Mean-Variance and achieves 100% win-rates against Teacher, Risk-Parity, and BNN-S on raw Sharpe. By contrast, BNN-S (row 3) records 80% against Teacher and 27% against Min-Variance and 0% against Mean-Var. The raw-Sharpe dominance of Mean-Variance must be interpreted alongside tail-risk analysis (Figure 11 ), which reveals Mean-Variance’s Pareto-dominated position for risk-averse investors [ Rockafellar and Uryasev, 2000 ] .
Deep reinforcement learning (DRL) frameworks for portfolio optimization have shown promise for their ability to learn allocation rules dynamically from market data. However, these models fail to account for fat-tailed returns, which characterize actual market behavior with more frequent extreme events. Furthermore, historical data is treated homogeneously, without accounting for temporal importance, leading models to fail during regime changes. We propose a new BAVAR-BLED algorithm that combines methods derived from Bayesian-Averaging Vector Autoregressive (BAVAR) and the Black-Litterman model using Elliptical Distributions (BLED) within a TD3 architecture. BAVAR captures a set of vector autoregressive representations that consider multi-scale temporal features, enabling adaptive allocation decisions based on regime-aware estimates of return expectations and dispersion matrices. These estimates serve as prior inputs to BLED, a model that uses Student's t-distributions, allowing for more realistic fat tail return estimates. The BAVAR-BLED algorithm uses transformer networks for view construction and CNNs for risk-aversion estimates, which modify dynamic allocation decisions based on market conditions. An evaluation of 29 Dow Jones Industrial Average constituents over a decade-long market period shows that BAVAR-BLED significantly outperforms state-of-the-art methods, achieving Sharpe and Sortino ratios of 1.72 and 2.70, respectively, and total returns of 57.26%.
The standard generalization bounds assume that the training and deployment distributions are the same, or are static, and don't consider regime switching environments where the ratio of calm vs crisis states is different. This paper proposes a framework that generalizes regime-aware models by quantifying the extra risk due to regime composition mismatch, when distribution shifts are Markov-switching. We obtain an exact decomposition, separating regime mismatch from regime sensitivity; we extend the bound to beta-mixing data using the effective sample size corrected for the spectral gap; and we show a minimax lower bound for synthetic data and on 25 years of global equity indices. The proposed penalty is an ex post realized generalization gap, whereas the training-only estimator does not show significant correlation: the feature geometry of crises can be detected, but not the temporal arrival. Thus, the framework is not a forecast machine. Forecasting the composition of the future regime is an open question in the rare cases of regime change.
Mean-variance portfolio optimization (MVO) is a central framework in data-driven asset management. A widely adopted approach is a two-stage framework that first predicts expected returns and then solves the optimization problem based on these predictions, with the predictive models trained by minimizing prediction errors. However, this objective of prediction is not aligned with the quality of the downstream portfolio decision. Decision-focused learning (DFL), which directly minimizes the downstream decision loss within the learning process, has thus emerged as a promising direction. However, existing DFL approaches to MVO rely on surrogate losses or constraint relaxations for tractability, creating a structural mismatch between predictive model training and the constrained MVO solved at evaluation. We propose a single-level optimization formulation that incorporates the Karush-Kuhn-Tucker (KKT) optimality conditions of the lower-level MVO into the upper-level learning problem. This formulation explicitly preserves the budget and short-sale constraints while remaining tractable for standard nonlinear optimization solvers. Rolling-window experiments on real-world ETF (Exchange Traded Funds) data across two asset universes with different correlation structures show that our method achieved the best performance on multiple investment metrics and also demonstrated performance improvement due to the proposed regularization.
Kensei Nosaka, Shunnosuke Ikeda, Yuichi Takano
University of Tsukuba, Tsukuba-shi, Ibaraki 305-8573, Japan