Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

Arishi Orra; Himanshu Choudhary; Manoj Thakur

cs.LGAug 16, 2026

Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

Authors: Arishi Orra, Himanshu Choudhary, Manoj Thakur

Organizations: School of Mathematical and Statistical Sciences Indian Institue of Technology Mandi Mandi, Himachal Pradesh, India, 175005

Abstract

Designing effective trading strategies using reinforcement learning remains challenging due to delayed and noisy rewards, poor exploration, and the difficulty of enforcing explicit risk constraints. In this work, we propose BRaG, a barycenter-based adversarial inverse reinforcement learning framework for stock trading that learns trading behavior from multiple heterogeneous expert strategies. BRaG aggregates expert demonstrations using a performance-weighted Wasserstein barycenter, yielding a stable pseudo-expert representation that captures shared structure across diverse trading styles. This representation is used to pretrain a trading policy via adversarial imitation learning, which alleviates unstable exploration during reinforcement learning. The pretrained policy is subsequently refined using reinforcement learning with true market rewards. To ensure risk-aware decision-making, BRaG incorporates control barrier functions that constrain action execution and regularize policy learning to satisfy drawdown limits. We evaluate the proposed approach on four major global equity markets, including the US, UK, Indian, and Taiwanese indices. Across all the markets, the proposed approach achieves stronger performance than both classical trading rules and recent deep reinforcement learning methods, while exhibiting more stable risk characteristics.

Figures & tables

Dataset	Metrics	Index	Buy-Hold	MVO	Random	CSMOM	TSMOM	Bollinger	MA Crossover	A2C	DDPG	VS-DRL	SRRS	RSHF	Adaptive	DREB	PPO	$\text{BRaG}^{\dagger}$
DJI	Cumulative Return	22.8053	30.561	19.649	18.678	24.241	25.315	25.625	31.926	37.192	38.268	36.959	12.275	35.708	24.023	42.522	35.594	53.977
	Annual Return	12.576	16.522	10.899	10.379	13.334	13.897	14.064	17.323	20.003	20.545	19.885	6.905	19.252	13.219	22.626	17.669	28.262
	Sharpe Ratio	0.883	1.117	0.976	0.713	0.661	0.914	0.814	1.062	1.067	1.165	1.193	0.457	1.289	0.947	1.405	1.164	1.472
	Max Drawdown	16.369	15.916	10.663	17.142	29.982	13.501	15.991	18.443	23.631	19.724	19.351	21.524	15.701	16.401	17.968	16.208	17.958
	Win Ratio	54.13	55.15	53.44	47.91	53.21	49.31	54.36	47.94	55.05	54.19	57.14	50.01	54.59	55.28	57.34	55.73	58.05
FTSE	Cumulative Return	20.443	8.604	23.597	-14.446	17.165	32.061	27.605	30.802	33.917	29.585	17.478	19.144	33.788	30.541	43.898	32.294	64.241

Table 1: Performance comparison of the proposed BRaG model against various baselines across the datasets.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hyperparameter	Range
PPO	Hidden Dimension	[2,512]
	Number of Layers	[1,8]
	Activation Function	[ReLU, Tanh, Sigmoid]
	Learning Rate	$[e^{-8},e^{-1}]$
	Dropout	[0,0.5]
	$\gamma$	[0.9,0.99]

Appendix

Table 2: Hyperparameter configuration and search space for the proposed BRaG model.

Panel of Figure 1 — Figure 1: Cumulative return trajectories of the proposed BRaG model and baseline strategies across the DJI, FTSE, Sensex, and TWII datasets during the out-of-sample trading period.

Dataset	Metrics	Random	A2C	DDPG	VS-DRL	SRRS	RSHF	Adaptive	DREB	PPO	$\text{BRaG}^{\dagger}$
DJI	Cumulative Return	18.68 $\pm$ 5.42	37.19 $\pm$ 3.41	38.27 $\pm$ 4.12	36.96 $\pm$ 3.85	12.28 $\pm$ 2.15	35.71 $\pm$ 2.94	24.02 $\pm$ 2.65	42.52 $\pm$ 3.10	35.59 $\pm$ 2.87	53.98 $\pm$ 1.95
	Annual Return	10.38 $\pm$ 3.15	20.00 $\pm$ 1.84	20.55 $\pm$ 2.21	19.89 $\pm$ 1.95	6.91 $\pm$ 1.12	19.25 $\pm$ 1.65	13.22 $\pm$ 1.43	22.63 $\pm$ 1.75	17.67 $\pm$ 1.52	28.26 $\pm$ 1.05
	Sharpe Ratio	0.71 $\pm$ 0.18	1.07 $\pm$ 0.12	1.17 $\pm$ 0.14	1.19 $\pm$ 0.13	0.46 $\pm$ 0.16	1.29 $\pm$ 0.11	0.95 $\pm$ 0.14	1.41 $\pm$ 0.09	1.16 $\pm$ 0.10	1.47 $\pm$ 0.06
	Max Drawdown	17.14 $\pm$ 3.85	23.63 $\pm$ 2.15	19.72 $\pm$ 2.84	19.35 $\pm$ 2.45	21.52 $\pm$ 3.12	15.70 $\pm$ 1.85	16.40 $\pm$ 2.05	17.97 $\pm$ 1.75	16.21 $\pm$ 1.64	17.96 $\pm$ 1.25
	Win Ratio	47.91 $\pm$ 2.45	55.05 $\pm$ 1.34	54.19 $\pm$ 1.65	57.14 $\pm$ 1.45	50.01 $\pm$ 1.85	54.59 $\pm$ 1.25	55.28 $\pm$ 1.55	57.34 $\pm$ 1.15	55.73 $\pm$ 1.15	58.05 $\pm$ 0.85
FTSE	Cumulative Return	-14.45 $\pm$ 6.15	33.92 $\pm$ 3.85	29.59 $\pm$ 4.15	17.48 $\pm$ 3.25	19.14 $\pm$ 3.65	33.79 $\pm$ 3.15	30.54 $\pm$ 2.95	43.90 $\pm$ 3.25	32.29 $\pm$ 3.14	64.24 $\pm$ 2.41

Appendix

Table 3: Performance comparison of the proposed BRaG model against various baselines across the datasets. Results are reported as mean

\pm

standard deviation over five independent runs.

Panel of Figure 2 — Figure 2: Distribution of Calmar and Sortino ratios across all models and datasets, illustrating comparative downside risk and risk-adjusted performance.

Figure 3: Effect of varying the number of expert trajectories used for barycenter construction on cumulative return and Sharpe ratio. Experts are added incrementally based on validation performance.

Figure 4: Effect of varying the maximum drawdown limit in the control barrier function on cumulative return and Sharpe ratio.

Barycenter weighting scheme	Cumulative Return $\uparrow$	Sharpe Ratio $\uparrow$	Maximum Drawdown $\downarrow$
Sharpe-based weighting	53.977	1.472	17.958
Equal weighting	48.502	1.281	20.133

Appendix

Table 4: Effect of expert-weighting schemes in the Wasserstein barycenter on the DJI dataset. Sharpe-based weighting assigns greater influence to experts with stronger risk-adjusted validation performance, whereas equal weighting assigns identical importance to all experts.

Component removed	Cumulative Return $\uparrow$	Sharpe Ratio $\uparrow$	Maximum Drawdown $\downarrow$
GAIL Pretraining	33.501	1.245	15.796
PPO Fine-tuning	42.378	1.335	16.953
CBF	52.951	1.258	25.874
None—Full BRaG	53.977	1.472	17.958

Appendix

Table 5: Component-wise ablation of BRaG on the DJI dataset. Each variant removes one major component from the full framework while retaining the remaining modules.

Expert Aggregation Method	Cumulative Return $\uparrow$	Sharpe Ratio $\uparrow$	Maximum Drawdown $\downarrow$
Best Single Expert	27.235	1.205	14.982
Uniform Expert Sampling	46.445	1.285	20.181
Sharpe-weighted Sampling	50.213	1.388	18.875
Full BRaG	53.977	1.472	17.958

Appendix

Table 6: Comparison of expert-demonstration aggregation strategies on the DJI dataset.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading
Aug 16, 2026Arishi Orra, Himanshu Choudhary, Manoj ThakurReinforcement Learning Representation Learning for RL
School of Mathematical and Statistical Sciences, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India
Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining
Sep 25, 2026Bicheng Wang, Xinyi ZhangAlgorithmic Trading Reinforcement Learning
Stanford University
PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading
Oct 1, 2026Duong Hien Chi Kien, Thanh Trung HuynhReinforcement Learning Risk-Sensitive RL
College of Engineering and Computer Science, VinUniversity Ha Noi, Viet Nam

Aug 16, 2026J/K move · Enter open · S save

Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading

Arishi Orra, Himanshu Choudhary, Manoj Thakur

School of Mathematical and Statistical Sciences, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India

Reinforcement Learning Representation Learning for RL Value Function Estimation

Open paper AskPDF arXiv

Reinforcement learning has gained increasing attention as a data-driven approach for stock trading. However, learning a policy that is both profitable and stable remains challenging due to non-stationary market behaviour and noisy reward signals. Auxiliary tasks are often used to improve representation learning and stabilize training, yet they are usually designed manually and depend heavily on prior assumptions about targets and prediction horizons. Such fixed designs may not remain suitable across changing market regimes. In this work, we propose a self-supervised framework that automatically discovers auxiliary tasks to support reinforcement learning for stock trading. The auxiliary tasks are formulated as General Value Functions so that their predictions enrich the learned state representation and assist policy optimization. The framework consists of two networks. The main network learns the trading policy along with the auxiliary predictions, while the secondary network generates the definitions of auxiliary tasks through learned cumulants and discount factors. These tasks are updated using a meta gradient mechanism that accounts for their long-term impact on trading performance and improves training stability. We evaluate the proposed approach across four major equity indices: DJI, FTSE, Sensex, and TAIEX. The empirical results demonstrate that automatically discovered auxiliary tasks lead to more robust learning and improved trading performance compared to existing baselines.

Abstract

Reinforcement Learning

Explore similar work

Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading

Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining

PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading