q-fin.GNMar 20, 2026

Decomposable Reward Modeling and Realistic Environment Design for Reinforcement Learning-Based Forex Trading

Authors: Nabeel Ahmad Saidd

Organizations: Dr APJ Abdul Kalam Technical University, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Applying reinforcement learning (RL) to foreign exchange (Forex) trading remains challenging because realistic environments, well-defined reward functions, and expressive action spaces must be satisfied simultaneously, yet many prior studies rely on simplified simulators, single scalar rewards, and restricted action representations, limiting both interpretability and practical relevance. This paper presents a modular RL framework designed to address these limitations through three tightly integrated components: a friction-aware execution engine that enforces strict anti-lookahead semantics, with observations at time t, execution at time t+1, and mark-to-market at time t+1, while incorporating realistic costs such as spread, commission, slippage, rollover financing, and margin-triggered liquidation; a decomposable 11-component reward architecture with fixed weights and per-step diagnostic logging to enable systematic ablation and component-level attribution; and a 10-action discrete interface with legal-action masking that encodes explicit trading primitives while enforcing margin-aware feasibility constraints. Empirical evaluation on EURUSD focuses on learning dynamics rather than generalization and reveals strongly non-monotonic reward interactions, where additional penalties do not reliably improve outcomes; the full reward configuration achieves the highest training Sharpe (0.765) and cumulative return (57.09 percent). The expanded action space increases return but also turnover and reduces Sharpe relative to a conservative 3-action baseline, indicating a return-activity trade-off under a fixed training budget, while scaling-enabled variants consistently reduce drawdown, with the combined configuration achieving the strongest endpoint performance.

Figures & tables

Figure 1: System architecture of the RL trading framework, linking configuration, data processing, training, deterministic backtesting, and reproducibility logging in one end-to-end pipeline.
IDActionPrimary legality preconditionExecution effect
0HOLDAlways legalMaintain current position; no new order submitted
1OPEN_LONGFlat position and sufficient free marginOpen long exposure at base lot size
2OPEN_SHORTFlat position and sufficient free marginOpen short exposure at base lot size
3PYRAMID_LONGExisting long position, scaling depth below cap, margin availableAdd to long position using configured pyramid increment
4PYRAMID_SHORTExisting short position, scaling depth below cap, margin availableAdd to short position using configured pyramid increment
5MARTINGALE_LONGExisting long position, martingale depth below cap, margin availableAdd long exposure after adverse movement using martingale scaling rule
Table 3: Extended action semantics and legality conditions (executed at opent+1open_{t+1} ).
Figure 2: Causality-correct environment step timing: the environment emits st\mathbf{s}_{t} , receives ata_{t} , execution is driven by market opent+1\mathrm{open}_{t+1} , portfolio state is marked at closet+1\mathrm{close}_{t+1} , reward engine returns only rtr_{t} to the environment, and the environment emits the full transition (st,at,rt,st+1)(\mathbf{s}_{t},a_{t},r_{t},\mathbf{s}_{t+1}) without any Reward →\rightarrow Agent bypass.
Figure 3: Reward taxonomy and computation flow for the RL trading environment, from agent–environment interaction through component aggregation and normalization to the final scalar reward.
Figure 4: Training-loop schematic with mask-aware interaction, complete transition construction, replay learning, and periodic target synchronization.
Figure 5: Multi-panel dashboard of endpoint metrics across retained experiment families, summarizing return, drawdown, risk-adjusted performance, and activity.
Figure 6: Reward-ablation comparison for variants r1 – r7 on EURUSD, showing a non-monotone profile with r7 as the strongest endpoint.
Figure 7: Training equity trajectories for r1 – r7 , highlighting non-monotone intermediate behavior and the strongest endpoint for r7 .
Figure 8: Equity-curve comparison of simplified (3-action) and extended (10-action) interfaces on EURUSD, showing a return–smoothness trade-off.
Figure 9: Training equity trajectories for scaling variants s1 – s4 on EURUSD, where scaling-enabled variants reduce maximum drawdown versus no scaling.
Figure 10: Equity trajectories for RL agents and benchmark strategies on EURUSD, with rule-based baselines as calibration anchors.
Figure 11: Multi-pair equity trajectories (EURUSD, GBPUSD, USDJPY, AUDUSD) for value-based agents under aligned simulator and reward settings.
Figure 12: Aggregate action distributions across training runs, showing broader action usage and higher activity in the extended interface.
Figure 13: Episode-level average pyramid depth (AvgPyr) and martingale depth (AvgMart) by scaling variant.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,668 papers

Explore similar work

CardsList
Mar 30, 2026

Realistic Market Impact Modeling for Reinforcement Learning Trading Environments

Reinforcement learning (RL) has shown promise for trading, yet most open-source backtesting environments assume negligible or fixed transaction costs, causing agents to learn trading behaviors that fail under realistic execution. We introduce three Gymnasium-compatible trading environments -- MACE (Market-Adjusted Cost Execution) stock trading, margin trading, and portfolio optimization -- that integrate nonlinear market impact models grounded in the Almgren-Chriss framework and the empirically validated square-root impact law. Each environment provides pluggable cost models, permanent impact tracking with exponential decay, and comprehensive trade-level logging. We evaluate five DRL algorithms (A2C, PPO, DDPG, SAC, TD3) on the NASDAQ-100, comparing a fixed 10 bps baseline against the AC model with Optuna-tuned hyperparameters. Our results show that (i) the cost model materially changes both absolute performance and the relative ranking of algorithms across all three environments; (ii) the AC model produces dramatically different trading behavior, e.g., daily costs dropping from 200kto200k to 8k with turnover falling from 19% to 1%; (iii) hyperparameter optimization is essential for constraining pathological trading, with costs dropping up to 82%; and (iv) algorithm-cost model interactions are strongly environment-specific, e.g., DDPG's OOS Sharpe jumps from -2.1 to 0.3 under AC in margin trading while SAC's drops from -0.5 to -1.2. We release the full suite as an open-source extension to FinRL-Meta.

Lucas Riera Abbade, Anna Helena Reali Costa

Escola Polit´ecnica Universidade de S˜ao Paulo

Algorithmic TradingRL Benchmarks
AskPDF
Mar 22, 2026

FinRL-X: An AI-Native Modular Infrastructure for Quantitative Trading

We present FinRL-X, a modular and deployment-consistent trading architecture that unifies data processing, strategy construction, backtesting, and broker execution under a weight-centric interface. While existing open-source platforms are often backtesting- or model-centric, they rarely provide system-level consistency between research evaluation and live deployment. FinRL-X addresses this gap through a composable strategy pipeline that integrates stock selection, portfolio allocation, timing, and portfolio-level risk overlays within a unified protocol. The framework supports both rule-based and AI-driven components, including reinforcement learning allocators and LLM-based sentiment signals, without altering downstream execution semantics. FinRL-X provides an extensible foundation for reproducible, end-to-end quantitative trading research and deployment. The official FinRL-X implementation is available at https://github.com/AI4Finance-Foundation/FinRL-Trading.

Hongyang Yang, Boyu Zhang, Yang She +2

AI4Finance Foundation

Algorithmic TradingQuantitative Finance
AskPDF
May 17, 2026

Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets

This study develops and evaluates a deep reinforcement learning framework for dynamic portfolio allocation across global equity markets. The Soft Actor-Critic algorithm is used to learn continuous portfolio weights within a Markov Decision Process, incorporating transaction costs, turnover penalties, and diversification constraints into the reward function. Five model configurations are compared, varying in reward formulation, policy structure (flat versus hierarchical Dirichlet), portfolio constraints, and temporal encoder (LSTM versus Transformer), and evaluated via walk-forward optimization across sixteen out-of-sample folds spanning 2003-2026 on the Nasdaq-100, Nikkei 225, and Euro Stoxx 50. Results show that RL strategies achieve competitive risk-adjusted performance primarily in the Euro Stoxx 50, where statistically significant abnormal returns are observed, but the central hypothesis is only partially confirmed: no strategy achieves statistically significant excess returns relative to Buy and Hold under HAC-robust inference across all markets. Regime analysis reveals that RL adds the most value during periods of elevated uncertainty, while ensemble aggregation across markets improves risk-adjusted performance and confirms the benefits of geographic diversification.

Kamil Kashif, Robert Ślepaczuk

Quantitative Finance Research Group Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland · Quantitative Finance Research Group Department of Quantitative Finance and Machine Learning Faculty of Economic Sciences University of Warsaw ul. Dluga 44-50 00-241 Warsaw Poland

Reinforcement LearningPortfolio OptimizationAlgorithmic Trading
AskPDF