Portfolio Reinforcement Learning with Scenario-Context Rollout

Vanya Priscillia Bendatu; Yao Lu

cs.AIFeb 27, 2026

Portfolio Reinforcement Learning with Scenario-Context Rollout

Authors: Vanya Priscillia Bendatu, Yao Lu

Organizations: National University of Singapore

Abstract

Market regime shifts induce distribution shifts that can degrade the performance of portfolio rebalancing policies. We propose macro-conditioned scenario-context rollout (SCR) that generates plausible next-day multivariate return scenarios under stress events. However, doing so faces new challenges, as history will never tell what would have happened differently. As a result, incorporating scenario-based rewards from rollouts introduces a reward--transition mismatch in temporal-difference learning, destabilizing RL critic training. We analyze this inconsistency and show it leads to a mixed evaluation target. Guided by this analysis, we construct a counterfactual next state using the rollout-implied continuations and augment the critic agent's bootstrap target. Doing so stabilizes the learning and provides a viable bias-variance tradeoff. In out-of-sample evaluations across 31 distinct universes of U.S. equity and ETF portfolios, our method improves Sharpe ratio by up to 76% and reduces maximum drawdown by up to 53% compared with classic and RL-based portfolio rebalancing baselines.

Figures & tables

Figure 1: Overview of our RL paradigm. SCR produces a conditional scenario distribution over next-day joint return vectors. The critic is trained using a bootstrap target that augments the realized continuation bootstrap with a counterfactual continuation.

Universe	Method	Sharpe $\uparrow$	Calmar $\uparrow$	AnnVol $\downarrow$	MaxDD $\downarrow$	Turnover
High-Vol	1/N	0.618 [0.563, 0.692]	0.912 [0.471, 2.114]	0.635 [0.356, 3.047]	0.445 [0.396, 0.493]	0.0000 [0.0000, 0.0000]
	Markowitz	0.561 [0.476, 0.699]	0.500 [0.317, 0.767]	0.527 [0.258, 0.780]	0.441 [0.411, 0.461]	0.0256 [0.0215, 0.0283]
	Inverse-Vol	0.630 [0.576, 0.695]	0.873 [0.411, 2.363]	0.589 [0.316, 2.350]	0.418 [0.404, 0.480]	0.0175 [0.0169, 0.0182]
	GMV (Ledoit-Wolf)	0.580 [0.533, 0.645]	0.850 [0.305, 1.397]	0.858 [0.518, 1.076]	0.412 [0.365, 0.449]	0.0261 [0.0216, 0.0305]
	PPO (Historical Replay)	0.615 [0.517, 0.719]	1.127 [0.445, 2.451]	0.569 [0.363, 3.674]	0.416 [0.385, 0.498]	0.298 [0.293, 0.299]
	BootRollout–PPO	0.598 [0.535, 0.729]	1.112 [0.502, 2.295]	0.647 [0.331, 3.798]	0.414 [0.367, 0.478]	0.2995 [0.2951, 0.2997]

Table 1: Evaluations and comparisons of portfolio rebalancing algorithms by universe group. Values are in median [Q1, Q3].

Episode Type	Macro Signature	Top Moves	Activity Period
Recurring	Geopolitical/Oil Shock $\mathrm{GPR}\uparrow,\ \mathrm{Brent}\uparrow,\ \mathrm{Energy}\uparrow$	$+$ ( XLP , XLV ); $-$ ( XLF , XLB )	2009–2023 (38 days; 14 in train)
Recurring	Policy Uncertainty $\mathrm{GPR}\downarrow,\ \mathrm{EPU}\uparrow,\ \mathrm{WTI}\downarrow$	$+$ ( XLK , XLV ); $-$ ( XLE , XLF )	2009–2021 (18 days; 16 in train)
Novel (Anomaly)	COVID-19 Liquidity Crisis No training-set channel match Flagged by novelty filter	Detected as novel episode	March 2020 (Consecutive days)

Table 2: Illustrative ShockLedger channels used in experiments.

Variant	Sharpe	Calmar	AnnVol	MaxDD	Turnover	Gap final	Resid-AUC
PPO (Historical Replay)	0.457 [0.256, 0.543]	0.222 [0.086, 0.335]	0.231 [0.188, 0.347]	0.381 [0.359, 0.414]	0.2320 [0.1600, 0.2983]	0.526 [0.304, 1.882]	0.217 [0.204, 0.231]
SCR–PPO–RewardOnly	0.506 [-0.054, 0.650]	0.364 [0.223, 0.584]	0.127 [0.107, 0.205]	0.181 [0.148, 0.330]	0.0050 [0.0049, 0.0051]	0.0003 [0.0001, 0.0016]	0.102 [0.054, 0.115]
SCR–PPO–NoCF	0.647 [0.571, 0.810]	0.561 [0.379, 0.833]	0.146 [0.110, 0.217]	0.184 [0.158, 0.198]	0.0045 [0.0042, 0.0049]	0.0001 [0.0000, 0.0012]	0.087 [0.077, 0.098]
SCR–PPO–Full	0.805 [0.635, 0.970]	0.737 [0.494, 0.984]	0.171 [0.122, 0.196]	0.179 [0.162, 0.194]	0.0046 [0.0044, 0.0049]	0.0001 [0.0000, 0.0003]	0.075 [0.061, 0.084]

Table 3: Ablation study of our solution. We report median values across multiple runs.

Figure 2: We stress test under shocks. The wealths are rebased to 1 at the window start. Top: COVID-19 sell-off window. Bottom: 2021–2022 macro shock window.

Figure 3: Scenario-to-real validation. Cumulative average daily return under SCR scenario scoring versus realized tape returns.

Figure 4: Critic stability under logged-tape mismatch. Bellman residual

\mathrm{resid}_{\ell_{2}}

versus training progress (mean

\pm

95% CI). Compared to PPO (Historical Replay), SCR–PPO–Full attains smaller residuals, and improved critic stability.

Setting	High-Vol	Low-Vol	General	Market-Proxy
$\beta_{cf}=0.0$	0.736	0.598	0.640	0.821
$\beta_{cf}=0.25$	0.529	0.529	0.716	0.823
$\beta_{cf}=0.5$	0.782	0.641	0.900	1.004
$\beta_{cf}=0.75$	0.518	0.509	0.671	0.838
$\beta_{cf}=1.0$	0.563	0.584	0.645	0.842

Table 4: Sensitivity to

\beta_{cf}

. We report median Sharpe here.

Portfolio Optimization

Portfolio optimization selects asset weights to balance expected return, risk, diversification, and trading constraints.

See all 63 papers

Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets
May 17, 2026Kamil Kashif, Robert ŚlepaczukReinforcement Learning Portfolio Optimization
Quantitative Finance Research Group Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland · Quantitative Finance Research Group Department of Quantitative Finance and Machine Learning Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland
Deep Reinforcement Learning for Reliability Based Bi-Objective Portfolio Optimization
Jul 7, 2026Sounaq Das, Tanmay Sen, Raghu Nandan Sengupta +1Multi-Objective Reinforcement Learning Risk-Sensitive RL
Indian Institute of Management Amritsar · Amritsar, India · SQC & OR Unit +7
Measure First, Then Allocate: Calibrated Rollout Budgets for MCP Agents in Reinforcement Learning
Oct 20, 2026Hieu Tran, Zonghai Yao, Sunjae Kwon +1RL for LLM Agents RL for Language Models
University of Massachusetts Amherst · University of Massachusetts Lowell

May 17, 2026J/K move · Enter open · S save

Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets

Kamil Kashif, Robert Ślepaczuk

Quantitative Finance Research Group Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland · Quantitative Finance Research Group Department of Quantitative Finance and Machine Learning Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland

Reinforcement Learning Portfolio Optimization Algorithmic Trading

Open paper AskPDF arXiv

This study develops and evaluates a deep reinforcement learning framework for dynamic portfolio allocation across global equity markets. The Soft Actor-Critic algorithm is used to learn continuous portfolio weights within a Markov Decision Process, incorporating transaction costs, turnover penalties, and diversification constraints into the reward function. Five model configurations are compared, varying in reward formulation, policy structure (flat versus hierarchical Dirichlet), portfolio constraints, and temporal encoder (LSTM versus Transformer), and evaluated via walk-forward optimization across sixteen out-of-sample folds spanning 2003-2026 on the Nasdaq-100, Nikkei 225, and Euro Stoxx 50. Results show that RL strategies achieve competitive risk-adjusted performance primarily in the Euro Stoxx 50, where statistically significant abnormal returns are observed, but the central hypothesis is only partially confirmed: no strategy achieves statistically significant excess returns relative to Buy and Hold under HAC-robust inference across all markets. Regime analysis reveals that RL adds the most value during periods of elevated uncertainty, while ensemble aggregation across markets improves risk-adjusted performance and confirms the benefits of geographic diversification.

Abstract

Portfolio Optimization

Explore similar work

Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets

Deep Reinforcement Learning for Reliability Based Bi-Objective Portfolio Optimization

Measure First, Then Allocate: Calibrated Rollout Budgets for MCP Agents in Reinforcement Learning