Dynamic Strategy Selection

Francesco Cordoni; Alessio Sancetta; Yuqian Zhao

cs.LGJul 7, 2023

Dynamic Strategy Selection

Authors: Francesco Cordoni, Alessio Sancetta, Yuqian Zhao

Organizations: Department of Economics, Royal Holloway University of London, Egham TW20 0EX, UK

Abstract

We consider the problem of dynamically selecting among competing strategies using observable predictors. To mitigate estimation noise and excessive switching, we regularise the estimated policy with a switching cost, which may exceed actual transaction costs when these are too small to provide sufficient regularisation. Since switching costs make current choices affect future decisions, we formulate the problem using dynamic programming. The optimal policy is estimated using a reinforcement learning algorithm based on iterated regressions, making the procedure simple to implement. We apply the method to monthly rotation among momentum, value, profitability, and defensive U.S. equity factors over a sample of about 60 years. Using a parsimonious set of state variables, the estimated policy outperforms both the myopic dynamic policy and the best fixed factor in hindsight in out-of-sample evaluation. Under weak conditions, we show theoretically that the decision rule estimated from empirical data is asymptotically consistent for the unknown optimal decision rule. Simulation results illustrate the finite-sample relevance of these asymptotic results.

Figures & tables

Figure 1: Cumulative Out-of-Sample Rewards. The figure plots the cumulative monthly rewards, in percent, of the RL, Greedy, and four fixed-factor policies. Results are reported when the states are Trend $+$ Cycle, with $\eta=0$ and a switching cost of 120 bps.

Panel of Figure 2 — Figure 2: State Dependent Factor Rotations. Results are reported when the states are Trend $+$ Cycle, with $\eta=3$ and a switching cost of 120 bps. Panel (a) displays the percentages of out-of-sample months in which the RL policy selects among the MOM, HML, RMW, and DEF strategies, conditional on the market states classified by the momentum turning points. The right sub-plot in Panel (a) shows the destination factor conditional on exiting momentum across different market states. $N$ denotes the number of out-of-sample months in each market state, whereas $n$ denotes the number of months in which the policy exits MOM in each market state. Percentages may not sum to 100 due to rounding. Panel (b) shows the factor selected by the RL policy over the out-of-sample period.

Figure S.1 : Cumulative Out-of-Sample Rewards. The figure plots the cumulative monthly rewards, in percent, of the RL, Greedy, and four fixed-factor policies. Results are reported when the states are Trend $+$ Cycle, with $\eta=0$ and a switching cost of 20, 80, and 120 bps.

Figure S.2 : Cumulative Out-of-Sample Rewards. The figure plots the cumulative monthly rewards, in percent, of the RL, Greedy, and four fixed-factor policies. Results are reported when the states are Trend $+$ Cycle, with $\eta=3$ and a switching cost of 20, 80, and 120 bps.

Panel of Figure 5 — Figure S.3 : State Dependent Factor Rotations. Results are reported when the states are Trend $+$ Cycle with $\eta=0$ . Panels (a) (c) (e) display the percentages of out-of-sample months in which the RL policy selects among the MOM, HML, RMW, and DEF strategies, conditional on the market states classified by the momentum turning points, for costs of 120, 80, and 20 bps, respectively. The corresponding right sub-plots show the destination factor conditional on exiting momentum across different market states. $N$ denotes the number of out-of-sample months in each market state, whereas $n$ denotes the number of months in which the policy exits MOM in each market state. Percentages may not sum to 100 due to rounding. The bottom panel shows the factor selected by the RL policy over the out-of-sample period.

Panel of Figure 6 — Figure S.4 : State Dependent Factor Rotations. Results are reported for the Trend $+$ Cycle state specification with $\eta=3$ . Panels (a) and (b) correspond to switching costs of 80 and 20 bps, respectively. Each panel reports the percentages of out-of-sample months in which the RL policy selects the MOM, HML, RMW, and DEF strategies, the destination factor conditional on exiting MOM across market states, and the factor selected by the RL policy over the out-of-sample period. For the results with a cost of 120 bps and $\eta=3$ , we refer to Figure 2 in the main text.

Figure S.5 : Boxplot of Monte Carlo Performance. For each information set, the figure reports the distributions of the RL and Greedy strategies across 500 replications for training-sample sizes of $n_{\mathrm{est}}=240$ , 600 and 1200. Each replication contains 500 out-of-sample observations. Boxes indicate the interquartile range. The oracle and strategy 1 are presented as benchmarks. Strategy 2 is omitted because its performance is similar to that of Strategy 1, while Strategies 3 and 4 are omitted because of their weaker performance.

Policy Learning

Policy learning develops decision rules that choose actions or interventions to maximize outcomes from data, feedback, or interaction.

See all 488 papers

Reinforcement learning to choose optimizers
Sep 1, 2026Martin van der Schelling, Deepesh Toshniwal, Miguel A. BessaLearning to Optimize
Department of Mechanical Engineering Delft University of Technology, Delft, the Netherlands · Institute of Applied Mathematics Delft University of Technology, Delft, the Netherlands · School of Engineering Brown University, Providence, RI, USA
Dynamic Objective Selection with Safeguards and LLM Oversight for Financial Decision-Making
Jun 2, 2026Keigo Sakurai, Takahiro Ogawa, Miki Haseyama +2Quantitative Finance Sequential Decision Making
Hokkaido University · Nomura Asset Management Co., Ltd. · Kobe University +1
Alpha-R1: Alpha Screening with LLM Reasoning via Reinforcement Learning
Dec 29, 2025Zuoyou Jiang, Li Zhao, Rui Sun +6Quantitative Finance Algorithmic Trading
Shanghai Jiao Tong University Shanghai, China · FinStep, StepFun Shanghai, China · StepFun Shanghai, China

Sep 1, 2026J/K move · Enter open · S save

Reinforcement learning to choose optimizers

Martin van der Schelling, Deepesh Toshniwal, Miguel A. Bessa

Department of Mechanical Engineering Delft University of Technology, Delft, the Netherlands · Institute of Applied Mathematics Delft University of Technology, Delft, the Netherlands · School of Engineering Brown University, Providence, RI, USA

Learning to Optimize

Open paper AskPDF arXiv

No single optimization method is uniformly best for all problems, and the most suitable optimizer choice can change during a run. Existing approaches that change optimizer during execution typically predetermine part of the strategy: the portfolio is restricted to one algorithm class, the switch occurs once at a fixed time, or the frequency of decisions is treated as a hyperparameter rather than a learned one. We introduce "Reinforcement Learning to Choose Optimizers", which formulates the optimization algorithm choice as a sequential decision-making problem. At each decision, a recurrent policy reads the current run state and decides both which optimizer should be used next and for how long. The portfolio includes both gradient-based and derivative-free optimizers, and each switch passes on the current best solution and a representative step size. A context proxy conditions a gating network over expert heads, and training employs a decoupled actor-critic whose return is expressed in the same empirical runtime distribution metric used at evaluation. Training tasks and portfolio are designed jointly so that no optimizer dominates. On unseen problems, the learned policy outperforms every portfolio optimizer at all but the smallest budgets, and it remains robust under distribution shift.

Abstract

Policy Learning

Explore similar work

Reinforcement learning to choose optimizers

Dynamic Objective Selection with Safeguards and LLM Oversight for Financial Decision-Making

Alpha-R1: Alpha Screening with LLM Reasoning via Reinforcement Learning