Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading

Arishi Orra; Himanshu Choudhary; Manoj Thakur

cs.LGAug 16, 2026

Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading

Authors: Arishi Orra, Himanshu Choudhary, Manoj Thakur

Organizations: School of Mathematical and Statistical Sciences, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India

Abstract

Reinforcement learning has gained increasing attention as a data-driven approach for stock trading. However, learning a policy that is both profitable and stable remains challenging due to non-stationary market behaviour and noisy reward signals. Auxiliary tasks are often used to improve representation learning and stabilize training, yet they are usually designed manually and depend heavily on prior assumptions about targets and prediction horizons. Such fixed designs may not remain suitable across changing market regimes. In this work, we propose a self-supervised framework that automatically discovers auxiliary tasks to support reinforcement learning for stock trading. The auxiliary tasks are formulated as General Value Functions so that their predictions enrich the learned state representation and assist policy optimization. The framework consists of two networks. The main network learns the trading policy along with the auxiliary predictions, while the secondary network generates the definitions of auxiliary tasks through learned cumulants and discount factors. These tasks are updated using a meta gradient mechanism that accounts for their long-term impact on trading performance and improves training stability. We evaluate the proposed approach across four major equity indices: DJI, FTSE, Sensex, and TAIEX. The empirical results demonstrate that automatically discovered auxiliary tasks lead to more robust learning and improved trading performance compared to existing baselines.

Figures & tables

Panel of Figure 1 — Figure 1: The two-network architecture of the proposed QUESTrader.

Hyperparameter	Range
Hidden Dimension	[2,512]
Number of Layers	[1,8]
Activation Function	[ReLU, Tanh, Sigmoid]
Learning Rate	[ $e^{-8}$ , $e^{-1}$ ]
Dropout	[0,0.5]
Gamma	[0.9,0.99]

Table 1: Hyperparameter configuration and search space for the proposed QUESTrader model.

Models	Annual return (%)	Cumulative return (%)	Sharpe Ratio	Max drawdown (%)	Calmar ratio	Sortino Ratio
Buy-Hold	12.857	16.097	0.884	10.433	1.054	1.066
DJI	8.234	10.258	0.724	9.331	0.882	1.040
MVO [ 21 ]	11.744	14.687	1.247	8.365	2.188	1.796
Random	0.982	1.213	0.650	1.759	0.834	1.524
VS-DRL [ 50 ]	14.548 $\pm$ 1.42	18.250 $\pm$ 1.76	0.993 $\pm$ 0.08	13.232 $\pm$ 0.91	1.099 $\pm$ 0.10	1.407 $\pm$ 0.12
SRRS [ 34 ]	16.069 $\pm$ 1.53	20.190 $\pm$ 1.89	1.424 $\pm$ 0.11	9.774 $\pm$ 0.72	2.371 $\pm$ 0.18	2.003 $\pm$ 0.16

Table 2: Performance comparison of QUESTrader and the baseline methods on the DJI dataset.

Models	Annual return (%)	Cumulative return (%)	Sharpe Ratio	Max drawdown (%)	Calmar ratio	Sortino Ratio
Buy-Hold	5.651	6.984	0.774	11.081	0.816	0.906
FTSE	9.828	12.140	0.833	10.183	0.896	1.117
MVO [ 21 ]	8.429	10.609	0.938	10.062	0.972	1.071
Random	-0.240	-0.301	-0.117	1.701	-0.141	-0.165
VS-DRL [ 50 ]	12.750 $\pm$ 1.15	16.184 $\pm$ 1.47	0.791 $\pm$ 0.07	14.289 $\pm$ 1.05	0.892 $\pm$ 0.08	1.248 $\pm$ 0.11
SRRS [ 34 ]	10.797 $\pm$ 1.02	13.674 $\pm$ 1.29	0.436 $\pm$ 0.05	30.851 $\pm$ 2.01	0.349 $\pm$ 0.04	0.778 $\pm$ 0.09

Table 3: Performance comparison of QUESTrader and the baseline methods on the FTSE dataset.

Models	Annual return (%)	Cumulative return (%)	Sharpe Ratio	Max drawdown (%)	Calmar ratio	Sortino Ratio
Buy-Hold	5.461	6.768	0.482	15.721	0.411	0.614
Sensex	6.663	8.175	0.538	15.757	0.422	0.741
MVO [ 21 ]	7.830	9.652	0.703	18.076	0.433	1.034
Random	-0.605	-0.739	-0.254	3.990	-0.151	-0.343
VS-DRL [ 50 ]	11.065 $\pm$ 1.04	13.686 $\pm$ 1.29	0.723 $\pm$ 0.06	17.156 $\pm$ 1.21	0.644 $\pm$ 0.06	1.252 $\pm$ 0.11
SRRS [ 34 ]	10.688 $\pm$ 0.99	13.214 $\pm$ 1.25	0.709 $\pm$ 0.06	10.650 $\pm$ 0.84	1.003 $\pm$ 0.09	1.050 $\pm$ 0.10

Table 4: Performance comparison of QUESTrader and the baseline methods on the Sensex dataset.

Models	Annual return (%)	Cumulative return (%)	Sharpe Ratio	Max drawdown (%)	Calmar ratio	Sortino Ratio
Buy-Hold	15.937	19.053	0.874	15.097	0.910	1.184
TWII	17.554	20.991	0.893	18.692	0.939	1.173
MVO [ 21 ]	18.007	21.468	1.738	8.065	2.173	2.171
Random	1.778	2.089	0.797	2.198	0.810	1.066
VS-DRL [ 50 ]	25.987 $\pm$ 2.21	31.173 $\pm$ 2.64	1.038 $\pm$ 0.09	21.581 $\pm$ 1.42	1.204 $\pm$ 0.11	1.519 $\pm$ 0.13
SRRS [ 34 ]	23.797 $\pm$ 2.05	28.498 $\pm$ 2.41	0.906 $\pm$ 0.08	24.492 $\pm$ 1.61	0.971 $\pm$ 0.09	1.276 $\pm$ 0.11

Table 5: Performance comparison of QUESTrader and the baseline methods on the TAIEX dataset.

Figure 2: Cumulative return trajectories of the proposed QUESTrader and baseline models on the DJI dataset.

Figure 3: Cumulative return trajectories of the proposed QUESTrader and baseline models on the FTSE dataset.

Figure 4: Cumulative return trajectories of the proposed QUESTrader and baseline models on the Sensex dataset.

Figure 5: Cumulative return trajectories of the proposed QUESTrader and baseline models on the TAIEX dataset.

Panel of Figure 6 — Figure 6: Risk–return scatter plot across all methods and markets.

Panel of Figure 7 — Figure 7: Action timelines: PPO baseline versus QUESTrader. Long/Short/Hold decisions overlaid on price. QUESTrader shows fewer whipsaws, earlier regime turns, and more disciplined holds.

Panel of Figure 8 — Figure 8: Ablation: effect of number of questions and unroll length on the Sharpe ratio.

Panel of Figure 9 — Figure 9: Ablation: effect of number of questions and unroll length on total return.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning
Aug 16, 2026Arishi Orra, Himanshu Choudhary, Manoj ThakurReinforcement Learning Finance
School of Mathematical and Statistical Sciences Indian Institue of Technology Mandi Mandi, Himachal Pradesh, India, 175005
CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach
Jul 17, 2026Andrei Neagu, Eeham Khan, Leila KosseimReinforcement Learning Deep Q-Learning
Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
Adaptive Regime-Aware Stock Price Prediction Using Autoencoder-Gated Dual Node Transformers with Reinforcement Learning Control
Mar 19, 2026Mohammad Al Ridhawi, Mahtab Haj Ali, Hussein Al OsmanTime Series Forecasting Finance
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada

Aug 16, 2026J/K move · Enter open · S save

Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

Arishi Orra, Himanshu Choudhary, Manoj Thakur

School of Mathematical and Statistical Sciences Indian Institue of Technology Mandi Mandi, Himachal Pradesh, India, 175005

Reinforcement Learning Finance Constrained RL Adversarial Imitation Learning

Open paper AskPDF arXiv

Designing effective trading strategies using reinforcement learning remains challenging due to delayed and noisy rewards, poor exploration, and the difficulty of enforcing explicit risk constraints. In this work, we propose BRaG, a barycenter-based adversarial inverse reinforcement learning framework for stock trading that learns trading behavior from multiple heterogeneous expert strategies. BRaG aggregates expert demonstrations using a performance-weighted Wasserstein barycenter, yielding a stable pseudo-expert representation that captures shared structure across diverse trading styles. This representation is used to pretrain a trading policy via adversarial imitation learning, which alleviates unstable exploration during reinforcement learning. The pretrained policy is subsequently refined using reinforcement learning with true market rewards. To ensure risk-aware decision-making, BRaG incorporates control barrier functions that constrain action execution and regularize policy learning to satisfy drawdown limits. We evaluate the proposed approach on four major global equity markets, including the US, UK, Indian, and Taiwanese indices. Across all the markets, the proposed approach achieves stronger performance than both classical trading rules and recent deep reinforcement learning methods, while exhibiting more stable risk characteristics.

Abstract

Reinforcement Learning

Explore similar work

Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach

Adaptive Regime-Aware Stock Price Prediction Using Autoencoder-Gated Dual Node Transformers with Reinforcement Learning Control