DeepSeekMath Meets Order Book: Group-Aware Policy Optimization for High-Frequency Directional Trading

Sayak Charabarty; Souradip Pal

cs.LGMay 25, 2026

DeepSeekMath Meets Order Book: Group-Aware Policy Optimization for High-Frequency Directional Trading

Authors: Sayak Charabarty, Souradip Pal

Organizations: Department of Computer Science Northwestern University Evanston, IL 60208, USA · School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47906, USA

Abstract

This paper studies reinforcement learning for high-frequency trading on limit order books by pairing an Order-Flow-based state model with policy-gradient methods. Instead of value-based RL techniques like tabular Q-learning, our approach deploys policy-based methods like vanilla PPO and DeepSeekMath-inspired variants like GRPO and GSPO, that use group-normalized updates and downside-aware shaping. On backtests with financial assets AMZN, AAPL, and GOOG under a simplified backtesting setup based on spread-scaled rewards, these new policies improve net average PnL, profitability, and drawdown over the Q-Learning baseline. Our results show that (1) Order-Flow signals are an adequate state for policy RL and (2) group-aware PPO surrogates are preferable over value-based baselines.

Explore similar work

OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
Jun 4, 2026Paavo Parmas, Yongmin Kim, Kohsei Matsutani +5
Plan Before You Trade: Inference-Time Optimization for RL Trading Agents
May 12, 2026Eun Go, Rohan Deb, Arindam BanerjeePortfolio Optimization Inference-Time Control
CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach
Jul 17, 2026Andrei Neagu, Eeham Khan, Leila KosseimDeep Reinforcement Learning Deep Q-Networks

Jun 4, 2026 · cs.LGJ/K move · Enter open · S save

OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation

Paavo Parmas, Yongmin Kim, Kohsei Matsutani, Shota Takashiro+4

The University of Tokyo

Policy-gradient methods usually optimize expected return, but many real world applications care about distributional properties of returns: tail risk, outlier robustness, or best-of-K discovery. We introduce OrderGrad, a family of likelihood-ratio and reparameterization gradient estimators for order-statistic objectives. OrderGrad optimizes finite-sample L-statistics, i.e., weighted averages of sorted rewards or costs, recovering objectives such as VaR, CVaR, trimmed means, medians, and top-m/best-of-K criteria by changing only the rank weights. For any fixed sample size and rank-weight vector, OrderGrad provides an unbiased gradient estimator for the corresponding order-statistic objective. The method is implemented as a simple reward transformation that can then be used in an otherwise standard policy-gradient or reparameterized update. We study the resulting estimator's variance behavior and evaluate it on tasks where mean optimization is mismatched to the deployment objective, including LLM math post-training and other tasks. OrderGrad provides a unified, plug-and-play route to risk-averse, robust, and exploratory learning. Code: https://github.com/paavo5/ordergrad

Open paper Ask about it PDF

Abstract

Explore similar work

OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation

Plan Before You Trade: Inference-Time Optimization for RL Trading Agents

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach