q-fin.MFApr 2, 2026

Reinforcement Learning for Speculative Trading under Exploratory Framework

Authors: Yun Zhao, Alex S. L. Tse, Harry Zheng

Organizations: Department of Mathematics, Imperial College, London SW7 2AZ, UK. · Department of Mathematics, University College London, London WC1H 0AY, UK.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study a speculative trading problem within the exploratory reinforcement learning (RL) framework of Wang et al. [2020]. The problem is formulated as a sequential optimal stopping problem over entry and exit times under general utility function and price process. We first consider a relaxed version of the problem in which the stopping times are modeled by the jump times of Cox processes driven by bounded, non-randomized intensity controls. Under the exploratory formulation, the agent's randomized control is characterized via the probability measure over the jump intensities, and their objective function is regularized by Shannon's differential entropy. This yields a system of the exploratory HJB equations and Gibbs distributions in closed-form as the optimal policy. Error estimates and convergence of the RL objective to the value function of the original problem are established. Finally, an RL algorithm is designed, and its implementation is showcased in a pairs-trading application.

Figures & tables

Figure 1: Value function V0(p)\mathcal{V}_{0}(p) under different (M,η)(M,\eta)
Figure 2: Value function V1(p,b)\mathcal{V}_{1}(p,b) under different (M,η)(M,\eta)
Figure 3: Optimal entry density πα,∗(λ;p)\pi^{\bm{\alpha},*}(\lambda;p) when η\eta is high
Figure 4: Optimal entry density πα,∗(λ;p)\pi^{\bm{\alpha},*}(\lambda;p) when η\eta is small
Figure 5: Optimal exit density πβ,∗(λ;p,b)\pi^{\bm{\beta},*}(\lambda;p,b) with M=50M=50 , η=10−5\eta=10^{-5}
Figure 6: Free boundary of pp as a function of bb with M=50M=50 , η=10−5\eta=10^{-5}
Figure 7: Value function V0(p)\mathcal{V}_{0}(p) under different θ\theta and σ\sigma
Figure 8: Comparison of value functions V0(p)\mathcal{V}_{0}(p) and V1(p,b)\mathcal{V}_{1}(p,b) between policy iteration and HJB with M=50,η=10−5M=50,\eta=10^{-5}
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

 loss(k)=1∑n=1N∑i=1I∑l=0L−1\mathds1{Jtln,i,k∈{0,1}}∑n=1N∑i=1I∑l=0L−1((δ0,ln,i,k)2+(δ1,ln,i,k)2).\displaystyle\texttt{loss}^{(k)}=\frac{1}{\sum_{n=1}^{N}\sum_{i=1}^{I}\sum_{l=0}^{L-1}\mathds{1}_{\{J_{t_{l}}^{n,i,k}\in\{0,1\}\}}}\sum_{n=1}^{N}\sum_{i=1}^{I}\sum_{l=0}^{L-1}\left((\delta_{0,l}^{n,i,k})^{2}+(\delta_{1,l}^{n,i,k})^{2}\right). 
Appendix
Algorithm 1 Offline Policy Iteration

Explore similar work

CardsList

  1. Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

    Sep 14, 2026Jin Ma, Gaozhan Wang, Jianfeng Zhang +1Reinforcement LearningStochastic Optimal Control

    Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027

  2. Continuous-time Optimal Stopping through Deep Reinforcement Learning

    Jun 16, 2026Cosmin Borsa, Michael LudkovskiQuantitative FinanceSequential Decision Making

    Department of Statistics & Applied Probability, UC Santa Barbara. Santa Barbara, CA 93106-3110, USA

  3. Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

    May 29, 2026Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada +4RL ExplorationPolicy Gradient Methods

    The University of Tokyo, Tokyo, Japan · Kobe University, Kobe, Japan · Kyoto University, Kyoto, Japan +3