Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining

Bicheng Wang; Xinyi Zhang

cs.LGSep 25, 2026

Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining

Authors: Bicheng Wang, Xinyi Zhang

Organizations: Stanford University

Abstract

Consistently profitable trading is difficult because equity markets are noisy, non-stationary, and only partially predictable from historical data. We benchmark five deep reinforcement learning (DRL) actor-critic methods: A2C, PPO, DDPG, TD3, and SAC, that learn trading actions end-to-end from market states, and compare them with a supervised price-forecasting baseline. Using daily data for 20 large-capitalization S&P 500 stocks from 2000 to 2020, enriched with trend-following technical indicators and log min-max scaling, we train on 2000-2018 and backtest on 2019-2020. Each agent is evaluated both when trained once and under forward retraining, in which it is retrained on all data available before each successive test window. DDPG achieves the highest annual return (55.5%), Sharpe ratio (1.38), and alpha (0.22), but also the highest market beta (1.24). TD3 and SAC offer a better risk-return balance, with Sharpe ratios of 1.37 and 1.33 and maximum drawdowns of about 25%. Forward retraining improves A2C, PPO, and SAC, leaves TD3 essentially unchanged, and reduces DDPG's annual return from 55.5% to 29.8%, consistent with TD3's greater robustness to hyperparameters. The forecasting baseline has the smallest maximum drawdown (9.6%) and the lowest beta (0.31), underscoring a trade-off between the higher returns of end-to-end DRL and the lower risk of forecast-driven strategies.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the two approaches. (a) An LSTM forecasts prices and a rule-based strategy turns the forecasts into positions. (b) A DRL agent interacts with a trading environment and learns trading actions end to end. Both are trained on 2000–2018 and backtested on 2019–2020.

Date	Open	High	Low	Close	Volume	Ticker
2000-01-03	0.936384	1.004464	0.907924	0.859423	535,796,800	AAPL
2000-01-03	16.812500	16.875000	16.062500	16.274673	7,384,400	ADBE
2000-01-03	81.500000	89.562500	79.046875	89.375000	16,117,600	AMZN
2000-01-03	25.125000	25.125000	24.000000	13.952057	13,705,800	BAC
2000-01-03	36.500000	36.580002	34.820000	35.299999	875,000	BRK-B

Table 1: Sample of the raw data. Close is the split- and dividend-adjusted closing price, which is why it can lie outside the unadjusted low–high range.

Ticker	boll_ub	$\cdots$	cci_10	dx_30	close_120_sma	close_120_ema
AAPL	0.9256	$\cdots$	$-66.67$	100	0.859	0.859
ADBE	0.9256	$\cdots$	$-66.67$	100	16.274	16.274
AMZN	0.9256	$\cdots$	$-66.67$	100	89.375	89.375
BAC	0.9256	$\cdots$	$-66.67$	100	13.952	13.952
BRK-B	0.9256	$\cdots$	$-66.67$	100	35.299	35.299

Table 2: Sample of the processed dataset (first trading day; intermediate columns omitted).

Model	Loss	MAE	MAPE	MSE ( $\times 10^{-6}$ )	MAE ${}_{\text{val}}$	MAPE ${}_{\text{val}}$	MSE ${}_{\text{val}}$ ( $\times 10^{-6}$ )
Baseline	1.052367	0.004578	1.052367	38	0.004099	0.972073	32
Dropout	2.371855	0.011330	2.371855	271	0.007442	1.535101	95
MSE	0.000336	0.006153	1.412530	336	0.004355	1.036787	38
Huber	0.000098	0.005407	1.253396	196	0.004077	0.974119	35
Log-cosh	0.000066	0.005286	1.226172	133	0.003798	0.898953	29
MAPE	0.935923	0.004015	0.935923	31	0.003789	0.888290	29

Table 3: Training and validation metrics of the LSTM variants. Loss is each model’s training objective, which is MAPE except for the MSE, Huber, and log-cosh models. Metrics are computed on scaled prices; the subscript val denotes validation. Best value in each metric column in bold.

Model	Annual return	Cumulative return	Sharpe ratio	Max drawdown	Alpha	Beta
LSTM	16.99%	36.85%	1.19	$-9.57\%$	0.09	0.31
LSTM (leveraged)	34.25%	80.22%	1.19	$-19.14\%$	0.20	0.61
A2C	20.47%	45.23%	0.82	$-31.11\%$	$-0.03$	1.02
A2C (retrained)	29.83%	68.73%	0.99	$-33.52\%$	0.03	1.14
PPO	27.89%	63.71%	1.09	$-25.33\%$	0.05	0.91
PPO (retrained)	32.16%	74.84%	1.30	$-22.84\%$	0.10	0.84

Table 4: Backtest performance on the 2019–2020 test set; the benchmark is the S&P 500. Retrained denotes forward retraining ( Section 4.2.3 ); other DRL rows are trained once. Best value in each column in bold (for maximum drawdown, the smallest decline); beta is a measure of market exposure and is not ranked.

Algorithmic Trading

Algorithmic trading uses computational models to generate signals, allocate portfolios, and execute orders across financial markets.

See all 144 papers

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach
Jul 17, 2026Andrei Neagu, Eeham Khan, Leila KosseimReinforcement Learning Deep Q-Learning
Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets
May 17, 2026Kamil Kashif, Robert ŚlepaczukReinforcement Learning Portfolio Optimization
Quantitative Finance Research Group Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland · Quantitative Finance Research Group Department of Quantitative Finance and Machine Learning Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland
Realistic Market Impact Modeling for Reinforcement Learning Trading Environments
Mar 30, 2026Lucas Riera Abbade, Anna Helena Reali CostaAlgorithmic Trading RL Benchmarks
Escola Polit´ecnica Universidade de S˜ao Paulo

Jul 17, 2026J/K move · Enter open · S save

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach

Andrei Neagu, Eeham Khan, Leila Kosseim

Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada

Reinforcement Learning Deep Q-Learning Policy Gradient Methods Deep Deterministic Policy Gradient Reward Design for RL Finance+1

Open paper AskPDF arXiv

This paper presents our system for Task 3 of the CLEF 2026 FinMMEval Lab, which requires daily long, flat, or short trading decisions for Bitcoin (BTC) and Tesla (TSLA) using news and historical market data. We formulate the problem as a discrete-action Markov Decision Process and compare four deep reinforcement learning algorithms: Policy Gradient (PG), Proximal Policy Optimization (PPO), Deep Q-Learning (DQL), and Deep Deterministic Policy Gradient (DDPG). The agents use technical indicators, cyclical calendar encodings, and daily news sentiment scores produced by LLaMA 3.2 1B. To reduce overfitting and align training with the objective of outperforming buy-and-hold, we introduce an alpha reward based on excess market return and randomize episode start dates. Hyperparameters are optimized with Ray Tune over 180 trials per algorithm-asset pair, with early stopping and model selection based on validation Sharpe ratio. On the CLEF Task 3 test set, DDPG achieves the strongest overall performance. DQL was selected a priori for the live endpoint because it obtained the highest validation Sharpe ratio, with selection performed without access to the test period. For TSLA, DDPG and DQL achieve cumulative returns of 54.96% and 52.62%, respectively, compared with 16.45% for buy-and-hold. For BTC, DDPG achieves a positive return of 1.58% while buy-and-hold declines by -34.27%. The results also reveal a substantial validation-to-test generalization gap, highlighting the difficulty of transferring policies selected in bull-market conditions to a bear-market regime.

Abstract

Algorithmic Trading

Explore similar work

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach

Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets

Realistic Market Impact Modeling for Reinforcement Learning Trading Environments