Organizations: Department of Computer Science, University of Colorado, Colorado Springs, 1420 Austin Bluffs Pkwy, Colorado Springs, CO 80918, USA · Department of Computer Science, University of Iowa, 101 Jessup Hall, Iowa City, IA 52242, USA
Reinforcement learning trading systems published in the academic literature overwhelmingly rely on price-aggregate state representations (OHLCV bars) or limit-order-book depth features, leaving microstructure pattern theories from the practitioner literature, namely Auction Market Theory and Market Profile, without a peer-reviewed computational instantiation. We present ViperQ, a reinforcement learning system whose state representation is built explicitly from Auction Market Theory primitives: Volume Point of Control, Value Area position, Low Volume Node flags, Cumulative Volume Delta divergence, and tape-velocity signatures, assembled into a 20-dimensional Z-normalised vector. Two Proximal Policy Optimisation agents are trained with a prospect theory-grounded asymmetric reward function that penalises losing holds at a magnitude consistent with Kahneman and Tversky's loss-aversion coefficient. Evaluated on a held-out twelve-month partition of institutional tick data the agents have never seen, ViperQ achieves +163.6% ROI on TSLA (-27.5% max drawdown, 27,019 trades) and +116.5% ROI on NVDA (-47.8% max drawdown, 12,892 trades) under zero leverage. The results establish Auction Market Theory features as a tractable structured input modality for sequential decision-making on financial time series and motivate further work on microstructure-aware policy learning.
Figures & tables
Figure 1: Matching engine: aggressive market flow consumes passive resting liquidity; price moves only when the resting size at the touch is fully consumed. ViperQ observes the consumption rate via CVD slope, tape velocity, and imbalance ratio (Section 3.2 ).
Figure 2: Auction Market Theory: the market oscillates between balance and imbalance. The agent targets the transition from Value Area A to Value Area B, remaining flat during balance phases and holding through breakouts.
Dim
Name
Construct
Block A: AMT / microstructure patterns
1
z_price_vwap
Z(price − VWAP), gravity to fair value
2
z_price_vpoc
Z(price − VPOC), distance to Volume Point of Control
3
dist_to_wall
Normalised position within 300 s high–low range
4
cvd_slope
60 s first-difference of CVD
5
cvd_divergence
Binary flag on 300 s price–CVD correlation <−0.5
Table 1: The 20-dimensional state vector st . Block A encodes Auction Market Theory and microstructure patterns; Block B is injected at runtime from the agent’s position; Block C supplies short-horizon memory.
Constraint
TSLA agent
NVDA agent
Min. hold (s)
300
300
Drawdown shock
−2.0 /step
−2.0 /step
Trend bonus
none
+0.10 /step ( U_{t}>\100$ )
Table 2: Asset-specific training configurations. The same architecture and state vector yield two distinct profitable policies. All constants listed here, with their selection mechanism, appear in Table 3 .
Figure 3: ViperQ system architecture: pattern extraction (Layers 1–3), policy inference (Layer 4, dual PPO), and risk-bounded execution (Layer 5 and output).
Element
Value
Mech.
Source / rationale
Reward function (Eqs. 1 – 2 , items (3)–(6))
α (PnL scaling)
0.01
VAL
PPO gradient stability
fsyn
$0.50/share
VAL
suppresses sub-tick scalping (§ 6.3 )
Hold gain branch
+0.001⋅Ut
VAL
disposition-effect tuning
Hold loss branch
−0.05−0.005∣Ut∣
TH
Tversky–Kahneman λ≈2.25 [ 26 ]
Trend bonus
+0.10 /step
VAL
held-winner reinforcement
Table 3: Master hyperparameter table. Mech. : VAL = selected on the validation slice of the training partition; CONV = industry convention for intraday proprietary trading, taken without tuning; TH = theory-derived with literature anchor; LIT = library default (Stable-Baselines3).
Metric
TSLA
NVDA
Final balance
$26,364.87
$21,652.86
PnL
+ $16,364.87
+ $11,652.86
ROI
+163.6%
+116.5%
Maximum drawdown
−27.5%
−47.8%
Total trades
27,019
12,892
Test-set bars
33.4 M
16.5 M
Table 4: ViperQ performance on the held-out test partition (final 20% of data, never seen during training). Initial capital 10,000perasset;zeroleverage;0.0002/share regulatory fees applied.
System
Algorithm
Asset / Horizon
ROI %
MDD %
DeepLOB [ 30 ]
CNN sup.
LOB / events
n.a.
n.a.
DRL-Trade [ 31 ]
DQN/PG/A2C
50 fut / daily
pos. Sharpe
n.r.
FinRL [ 29 ]
Ensemble
Dow 30 / daily
≈30
−10.2
PPO-Exec [ 15 ]
PPO/LSTM
LOB / per-step
beats TWAP
n.r.
TDQN [ 23 ]
DQN
30 eq / daily
beats pass.
n.r.
HFT-PPO [ 1 ]
PPO
INTC / events
stable +
n.r.
Table 5: ViperQ vs. recently published RL trading systems with retrievable performance numbers. Typical published range [ 20 ] : ROI [5%,40%] , MDD [−5%,−25%] . “n.r.” = not reported; “n.a.” = not applicable.
Figure 4: ViperQ in the ROI/MDD plane against published RL trading systems with concrete numbers. The shaded region marks the typical published range surveyed in [ 20 ] .
Department of Computer Science Northwestern University Evanston, IL 60208, USA · School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47906, USA