When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game
Authors: Siu Tung Wong, Carlo Campajola
Organizations: Institute of Finance and Technology, University College London · Institute of Finance and Technology University College London London, United Kingdom · UZH Blockchain Center · UZH Blockchain Center Zürich, Switzerland
Reinforcement learning (RL) is increasingly used for financial optimal-control problems when complex dynamics make analytical strategies difficult to obtain. There are financial mathematics literactures which provides many solved models whose equations and controls could evaluate and guide learning; we ask whether RL can exploit these results. We place a proximal policy optimisation (PPO) agent in an analytically solved continuous-time broker--trader game. PPO replaces the broker and chooses its trading speed while interacting with an informed trader and stochastic uninformed order flow. We derive a finite-step reward from the broker's continuous-time payoff and verify its discrete implementation through grid refinement and an exact one-step identity. With zero uninformed flow, a validation-selected PPO--FFNN approaches the reference action. With stochastic uninformed flow, the tested PPO--FFNN and PPO--LSTM remain inaccurate, although supervised learning confirms that their actors can represent the action. Monte Carlo diagnostics show that their critics do not reliably rank nearby actions; potential-based reward shaping also gives no reliable improvement. Under partial information, a causal certainty-equivalent controller based on the broker's observable history remains close to the reference, while PPO has larger errors and lower payoffs. Finally, we freeze the analytical policy and train PPO to adjust it after the execution cost changes. Halving the cost yields a repeatable improvement that closes 2.22% of the gap to the changed-cost reference. The analytical solution therefore provides both a benchmark for diagnosing RL and a useful starting policy for adaptation.
Figures & tables
Figure 1. Information and feedback for one broker decision. The policy maps the full- or partial-information observation to a trading rate, which is clipped to [−100,100] before the environment updates the state and reward. Red variables are hidden under partial information; past informed flow ηk is reconstructible at decision k+1 . The reference action ( νkref ) is evaluated at every PPO state. A three-column diagram shows the broker's observed and hidden information, the observation entering an actor, the actor action driving the environment, and the reference action νkref recalculated at the state reached by PPO.
Setting
PPO–FFNN
PPO–LSTM
Analytical policy with PPO adjustment
Actor / critic
Actor and critic each use two fully connected 64-unit ReLU hidden layers
Actor and critic each use 24-unit LSTM and a fully connected 64-unit Tanh hidden layer
100 critic-only warm-up and 800 joint updates; four environments × 1,024 steps; actor/critic rates 2×10−5/2×10−4 ; γ=1 , advantage-estimation λ=0.95 , clip 0.1 , batch 1,024, ten epochs
Training / selection
4×106 transitions; five training seeds; activation and checkpoints selected by finite-grid payoff on separate validation paths
Five training seeds; checkpoints at updates 50, 100, 200, 400, and 800; the highest mean payoff on 50 common validation paths is selected; final evaluation on 100 previously unused paths
Supervised diagnostic
ReLU FFNN and Tanh LSTM with the same actor shapes as PPO; Adam at 10−3 , 100 epochs; 40/20/40 analytical train/validation/test episodes
Not used
Table 1. Networks and training settings. The reported zero-flow FFNN uses an initial log standard deviation of 3; the other FFNN and LSTM experiments use 0. Checkpoints are selected by mean payoff on 50 separate validation paths. A compact comparison of the actor and critic architectures, PPO settings, training procedures, and analytical prior used by the feedforward, recurrent, and analytical-policy-adjustment controllers.
Δt
Steps N
LQ loss
Action RMSE
10−3
1,000
3.462×10−7
0.016835
5×10−4
2,000
8.681×10−8
0.008468
2×10−4
5,000
1.383×10−8
0.003389
Table 2. Summary of Blocks A–D. Action RMSE is the root mean squared difference between the broker action and the stated reference action ( νkref ). (a) compares the continuous analytical action with νkref at three time steps. (b) compares full-information PPO with νkref when ξ=0 . (c) evaluates the supervised FFNN and LSTM on analytical test states and then when their actions determine subsequent states. (d) reports payoff, payoff shortfall from the clipped reference, and action RMSE when σξ=100 . In (b) and (d), the reference is recalculated at each state reached by the controller. Brackets are 95% intervals across five training seeds; the LSTM entry 11.85–339.70 is the five-seed range. Higher payoff is better; lower LQ loss, payoff shortfall, and RMSE are better.
Figure 2. Trajectories and PPO diagnostics. Panels (a)–(b) show the zero-flow experiment and (c)–(d) the change from a=0.0012 to 0.0006 . Lines are means and bands span the 5th–95th percentiles; (a)–(b) use signed symmetric-log vertical scales. Panel (e) uses separate payoff scales and normalises progress by each training budget; markers denote checkpoints. Panels (f)–(g) show losses smoothed over 25 updates and averaged across five training seeds. PPO–FFNN/LSTM are from Block B and “Analytical policy + PPO” from Section 5. The dotted line marks the end of its 100-update critic warm-up. Seven panels show zero-flow broker trajectories, lower-cost misspecification trajectories, validation broker payoff during learning, PPO actor loss, and critic value loss.
Training reward / input
JΔtB
Action RMSE
EV
Corrected / with Sk
−5.75±4.82
19.76±4.15
.93±.07
First order / with Sk
−4.27±5.59
17.95±4.63
.79±.15
Corrected ×100 / with Sk
−34.99±.78
31.01±.19
.42±.35
Corrected / without Sk
−6.84±7.79
19.13±5.66
.84±.15
Table 3. Full-information ReLU FFNN ablations at σξ=100 . JΔtB is evaluated using Equation ( 12 ), and action RMSE is calculated using νkref . Explained Variance (EV) is the critic’s value predictions at the selected checkpoint. Relative to the corrected baseline, paired payoff changes (95% intervals) are 1.48[−7.34,10.31] without the Δt2 term, −29.24[−33.93,−24.54] after multiplying rewards by 100, and −1.09[−8.56,6.37] without Sk . Higher payoff and EV, and lower RMSE, are better. Broker payoff and action error for PPO trained with the corrected or first-order reward, with a fixed reward scaling, and without the absolute price input, together with in-sample critic explained variance.
Figure 3. Learning diagnostics. Left and centre: each point compares a+ and a− at one of 120 test states. The horizontal axis is their difference in cumulative reward estimated by Monte Carlo; the vertical axis is the difference predicted using the immediate reward and critic value of the next state. Each point is (ΔQkMC,ΔQθ,k) , where both differences subtract the payoff of a− from a+ . In the top-right region, both estimates prefer a+ ; in the bottom-left region, both prefer a− . In the top-left region, Monte Carlo prefers a− but the critic predicts a+ , while in the bottom-right region, Monte Carlo prefers a+ but the critic predicts a− . Pearson correlation measures whether the critic difference increases with the Monte Carlo difference, while Spearman correlation measures whether the critic ranks the action comparisons in the same order as Monte Carlo. The pooled Pearson and Spearman correlations are 0.216 and 0.053, and sign accuracy is 51.7%. Horizontal bars are 95% Monte Carlo intervals; open points indicate decisions where a+ and a− appear equally good within this uncertainty. Without including open points the remaining 71 points show a sign accuracy is 49.3%. Right: each seed’s payoff shortfall from the clipped reference, with σξ=100 ; diamonds and bars show the mean and 95% Student- t interval. Correlations closer to one and higher sign accuracy are better; lower payoff shortfall is better. The left panel shows weak association between critic and Monte Carlo local action preferences, the centre panel enlarges the dense region near zero, and the right panel shows five seed-level payoff shortfalls for full- and partial-information FFNN and LSTM policies.
atrue
Reference
Frozen
PPO adjusted
Change (95% CI)
0.0006
0.53010
0.29632
0.30151
0.00519[0.00296,0.00743]
0.0012
0.06882
0.06882
0.06901
0.00019[−0.00041,0.00079]
0.0018
−0.15783
−0.16164
−0.16145
0.00019[−0.00034,0.00071]
Table 4. Finite-grid payoff after changing execution cost. “Reference” is recalculated in Enew , while “frozen” uses E0 . Brackets are 95% Student- t intervals over five training seeds after averaging 100 common paths. Higher payoff and change are better. Mean broker payoff for the changed-cost reference, unchanged analytical policy, and PPO-adjusted policy at three execution costs.
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
Proximal Policy Optimization (PPO) has become the predominant algorithm for on-policy reinforcement learning due to its scalability and empirical robustness across domains. However, there is a significant disconnect between the underlying foundations of trust region methods and the heuristic clipped objective used in PPO. In this paper, we bridge this gap by introducing the Bounded Ratio Reinforcement Learning (BRRL) framework. We formulate a novel regularized and constrained policy optimization problem and derive its analytical optimal solution. We prove that this solution ensures monotonic performance improvement. To handle parameterized policy classes, we develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL. We further establish a lower bound on the expected performance of the resulting policy in terms of the BPO loss function. Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM). We additionally extend BPO to Group-relative BPO (GBPO) for LLM fine-tuning. Empirical evaluations of BPO across MuJoCo, Atari, and complex IsaacLab environments (e.g., Humanoid locomotion), and of GBPO for LLM fine-tuning tasks, demonstrate that BPO and GBPO generally match or outperform PPO and GRPO in stability and final performance.
Yunke Ao, Le Chen, Bruce D. Lee +5
ETH Zurich · MPI for Intelligent Systems · University of Alberta +2
In this paper, we investigate whether a model-free RL agent can identify and exploit price manipulation opportunities more effectively than a traditional model-based approach that assumes correct specification of the data-generating process but relies on noisy parameter estimates. We consider a single-asset market in which prices evolve according to an Almgren-Chriss framework with non-linear permanent impact and linear temporary impact. We first establish the existence of price-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information. We then compare two finite-sample learning approaches: a model-based procedure that estimates impact parameters from simulated execution data and an agnostic RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data. For intermediate volatility, the RL agent successfully discovers profitable manipulative strategies without explicit knowledge of the underlying model, even when training data are quite limited. More importantly, RL consistently outperforms the model-based approach when parameter estimates are affected by sampling error, despite the latter benefiting from the correct model specification. For large volatility, all methods are unable to identify manipulation opportunities, while for small volatility, the model based approach outperforms RL. These findings highlight both the effectiveness of RL in complex control problems and the risks associated with deploying learning algorithms in financial markets without appropriate safeguards.
Ioanna-Yvonni Tsaknaki, Andrea Macrì, Fabrizio Lillo
Scuola Normale Superiore, Piazza dei Cavalieri 7, Pisa 56126, Italy.
This paper studies reinforcement learning for high-frequency trading on limit order books by pairing an Order-Flow-based state model with policy-gradient methods. Instead of value-based RL techniques like tabular Q-learning, our approach deploys policy-based methods like vanilla PPO and DeepSeekMath-inspired variants like GRPO and GSPO, that use group-normalized updates and downside-aware shaping. On backtests with financial assets AMZN, AAPL, and GOOG under a simplified backtesting setup based on spread-scaled rewards, these new policies improve net average PnL, profitability, and drawdown over the Q-Learning baseline. Our results show that (1) Order-Flow signals are an adequate state for policy RL and (2) group-aware PPO surrogates are preferable over value-based baselines.
Sayak Charabarty, Souradip Pal
Department of Computer Science Northwestern University Evanston, IL 60208, USA · School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47906, USA