Market-making strategies in real limit order book markets face substantial model uncertainty and regime-shift risk. Existing adversarial reinforcement learning approaches improve robustness by formulating the Avellaneda--Stoikov market-making problem as a zero-sum game between a market maker and an environmental adversary. However, these approaches typically rely on Poisson order arrivals and neglect trade-induced price impact, limiting their ability to capture important high-frequency market microstructure effects such as clustered order flow, self-excitation, and post-trade price feedback. We extend adversarial reinforcement learning for market making to a more complex environment with Hawkes self-exciting order arrivals and trade-induced price impact. To mitigate the increased non-stationarity introduced by the expanded regime space, we incorporate an LSTM module that explicitly models the temporal structure of recent observations. We further characterize the equilibrium properties of the proposed framework through both game-theoretic analysis and numerical experiments, and introduce a robustness evaluation protocol focused on improvements in the left tail of the return distribution. Experimental results across a range of market regimes show that the proposed method achieves improved left-tail performance in most complex microstructure environments. In particular, the gains are pronounced in regimes with strong Hawkes excitation and low-to-moderate price impact. Bootstrap tests provide no evidence that these improvements are obtained through a stronger terminal directional inventory bias. These results suggest that combining adversarial training with temporal state representation can improve the robustness of reinforcement-learning-based market-making strategies under order-flow self-excitation, price impact, and regime uncertainty.
Figures & tables
Market dynamic / mechanism
ARLMM
Proposed method
Mid-price drift ( b )
Yes
Yes
Baseline liquidity ( A )
Yes
Yes
Order-book depth / execution sensitivity ( k )
Yes
Yes
Poisson arrivals
Yes
Special case
Hawkes self-exciting arrivals (κH,γH)
No
Yes
Trade-induced price impact (ξ)
No
Yes
Table 1: Comparison of market-microstructure mechanisms.
Environment
Regime parameters
Method
Terminal wealth
Sharpe
CVaR 10%
CVaR 30%
Terminal inventory
G1_std
(0, 140, 1.5, 60, 0, 0)
Fixed
14.9841 ± 6.9313
2.161794
3.888963
7.313555
-1.2930 ± 0.7847
Random
20.9421 ± 7.1667
2.922123
9.369169
12.935652
-1.4750 ± 0.7402
ARLMM
19.7519 ± 6.9503
2.841865
8.172405
11.832854
-0.5350 ± 0.7612
Ours
27.0037 ± 7.5168
3.592460
14.633210
18.339891
-0.1490 ± 1.0526
G6_hawkes_only
(0, 140, 1.5, 60, 20, 0)
Fixed
21.0198 ± 8.2750
2.540161
7.424113
11.704850
-1.3000 ± 0.8029
Random
29.9129 ± 8.7390
3.422921
15.247651
20.187225
-1.4360 ± 0.7553
Table 2: Representative test environments. G1 is the standard environment; G6 and G11 are Hawkes-dominated; G13 and G14 combine high drift and execution pressure; G18 and G19 are extreme/compound regimes. The regime-parameter vector is ordered as (b,A,k,κH,γH,ξ) . Complete results are reported in the appendix.
Figure 1: Kernel density estimates of terminal wealth in G19.
Figure 2: CVaR10% heatmaps over (γH,ξ) for Ours and ARLMM.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Environment
Regime parameters
Method
Terminal wealth
Sharpe
CVaR 10%
CVaR 30%
Terminal inventory
G1_std
(0, 140, 1.5, 60, 0, 0)
Fixed
14.9841 ± 6.9313
2.161794
3.888963
7.313555
-1.2930 ± 0.7847
Random
20.9421 ± 7.1667
2.922123
9.369169
12.935652
-1.4750 ± 0.7402
ARLMM
19.7519 ± 6.9503
2.841865
8.172405
11.832854
-0.5350 ± 0.7612
Ours
27.0037 ± 7.5168
3.592460
14.633210
18.339891
-0.1490 ± 1.0526
G2_drift_up
(5, 140, 1.5, 60, 0, 0)
Fixed
9.8071 ± 6.8781
1.425851
-1.625236
1.927832
-1.2750 ± 0.7576
Random
15.2982 ± 7.7047
1.985568
2.642664
6.452767
-1.4040 ± 0.7108
Appendix
Table 3: Complete test results across all 19 evaluation environments. The regime-parameter vector is ordered as (b,A,k,κH,γH,ξ) .
Market making under realistic order-flow dynamics is challenging due to the clustered, self-exciting nature of order arrivals, which are naturally modeled by non-Markovian Hawkes processes. We propose a model-free reinforcement learning (RL) framework for market making with general Hawkes order-flow dynamics. We first establish non-asymptotic performance bounds that quantify the loss incurred by approximating the infinite-dimensional Hawkes state using a finite collection of exponentially weighted features. Guided by this analysis, we introduce the Hawkes-Structured Network (HSN), a novel neural architecture that encodes historical order flow through a learnable mixture-of-exponentials filter. HSN is trained end-to-end using model-free RL, jointly optimizing the feature representation and quoting policy directly from reward feedback. Experiments on single- and multi-asset market-making problems with mixture-of-exponentials, power-law, and regime-switching kernels demonstrate that HSN outperforms general-purpose neural architectures, achieving higher trading rewards while implicitly recovering the underlying order-flow intensities.
Fernando Acero, Philipp Plank, Yufei Zhang
Department of Mathematics, Imperial College London, United Kingdom
We describe an adaptive market-making architecture that preserves the analytical structure of the Avellaneda--Stoikov framework while introducing a successor measure-style adaptation mechanism. In our paper we keep Avellaneda--Stoikov fast Hamilton--Jacobi--Bellman structure and make it adaptive to changing market regimes and trading objectives. The central idea is to separate market dynamics from the trading objective. The market state determines a low-dimensional set of Avellaneda--Stoikov parameters, while recent realized rewards determine a low-dimensional objective vector. The HJB forward map then converts this objective into optimal bid and ask quotes through a scalarization of future reward features.
We study online learning in market making, where, at each round, a market maker posts bid and ask prices before observing the market price and the private valuation of an incoming trader. In this setting, Maran et al. 2026 introduce a feedback model motivated by limit order books, in which the trader's valuation is revealed only if no transaction occurs. Assuming that trader valuations are drawn i.i.d. from an unknown distribution while market prices are chosen adversarially, they establish an expected regret bound of O(T2/3). In this work, we improve upon this guarantee by establishing a high-probability regret bound of O(T). As a warm-up, we first consider the full-feedback setting. We introduce a discretization of the bid-ask space based on two coupled grids and combine it with Hedge to achieve the desired regret rate. Building on these ideas, we then address the substantially weaker feedback induced by a limit order book and develop an algorithm that achieves the same guarantee. Finally, we investigate the limits of learnability in fully adversarial environments, where the valuations may vary arbitrarily as well. Perhaps surprisingly, we show that when both market prices and trader valuations are chosen adversarially, sublinear regret is impossible even under full feedback, thereby motivating our stochastic assumption on the valuations.
Maria Elena Vischi, Francesco Emanuele Stradi, Alberto Marchesi