Reinforcement Learning with Hawkes-Structured Networks for Market Making

Fernando Acero; Philipp Plank; Yufei Zhang

cs.LGOct 12, 2026

Reinforcement Learning with Hawkes-Structured Networks for Market Making

Authors: Fernando Acero, Philipp Plank, Yufei Zhang

Organizations: Department of Mathematics, Imperial College London, United Kingdom

Abstract

Market making under realistic order-flow dynamics is challenging due to the clustered, self-exciting nature of order arrivals, which are naturally modeled by non-Markovian Hawkes processes. We propose a model-free reinforcement learning (RL) framework for market making with general Hawkes order-flow dynamics. We first establish non-asymptotic performance bounds that quantify the loss incurred by approximating the infinite-dimensional Hawkes state using a finite collection of exponentially weighted features. Guided by this analysis, we introduce the Hawkes-Structured Network (HSN), a novel neural architecture that encodes historical order flow through a learnable mixture-of-exponentials filter. HSN is trained end-to-end using model-free RL, jointly optimizing the feature representation and quoting policy directly from reward feedback. Experiments on single- and multi-asset market-making problems with mixture-of-exponentials, power-law, and regime-switching kernels demonstrate that HSN outperforms general-purpose neural architectures, achieving higher trading rewards while implicitly recovering the underlying order-flow intensities.

Figures & tables

	Single-asset, mix-of-exp.		Multi-asset, exp.	Single-asset, fractional		Regime-switching, mix-of-exp.
Architecture	$n{=}0.75$	$n{=}0.90$	$\mathit{cs}{=}0.7$	$\varepsilon{=}0.50$	$\varepsilon{=}0.15$	$\mathit{sr}{=}2$	$\mathit{sr}{=}4$
HSN	$\mathbf{0.0171}_{{\pm.0007}}$	$\mathbf{0.0208}_{{\pm.0004}}$	$\mathbf{0.0548}_{{\pm.0007}}$	$\mathbf{0.0217}_{{\pm.0002}}$	$\mathbf{0.0216}_{{\pm.0004}}$	$\mathbf{0.0561}_{{\pm.0009}}$	$\mathbf{0.0451}_{{\pm.0006}}$
Simple MLP	$0.0090_{{\pm.0000}}$	$0.0144_{{\pm.0000}}$	$0.0349_{{\pm.0003}}$	$0.0125_{{\pm.0000}}$	$0.0093_{{\pm.0000}}$	$0.0339_{{\pm.0002}}$	$0.0286_{{\pm.0001}}$
MLP+EMA	$0.0110_{{\pm.0001}}$	$0.0177_{{\pm.0001}}$	$0.0349_{{\pm.0003}}$	$0.0163_{{\pm.0003}}$	$0.0127_{{\pm.0003}}$	$0.0389_{{\pm.0003}}$	$0.0342_{{\pm.0003}}$
MLP+frame-stack	$\mathbf{0.0184}_{{\pm.0006}}$	$\mathbf{0.0211}_{{\pm.0004}}$	$0.0470_{{\pm.0006}}$	$0.0206_{{\pm.0003}}$	$0.0163_{{\pm.0002}}$	$0.0505_{{\pm.0004}}$	$0.0428_{{\pm.0005}}$
GRU	$0.0094_{{\pm.0001}}$	$0.0155_{{\pm.0002}}$	$0.0450_{{\pm.0010}}$	$0.0168_{{\pm.0002}}$	$0.0148_{{\pm.0001}}$	$0.0443_{{\pm.0005}}$	$0.0338_{{\pm.0004}}$

Table 1. Mean cumulative reward

\hat{V}

across 30 seeds (continuous actions, parameter-matched architectures). Subscripts denote SEM. Bold entries are the column maximum. Bottom row: relative gap of HSN vs the strongest competing baseline in each column, shown where the difference is statistically significant (***

p<0.001

, **

p<0.01

, Welch’s

t

-test); “tie” denotes no significant difference (

p>0.05

). Column groups: single-asset with exponential Hawkes kernel (branching ratio

n

); multi-asset with exponential Hawkes kernel and cross-asset excitation (cross-strength

\mathit{cs}

); single-asset with fractional Hawkes kernel (tail index

\varepsilon

); single-asset with regime-switching exponential Hawkes kernel (switch rate

\mathit{sr}

Panel of Figure 1 — Figure 1. Distribution of final $\hat{V}$ across 30 seeds for two intensity-sensitive configurations of Table 1 . Boxes span the interquartile range (median line) and whiskers show min/max. Architectures are ordered by feature-extractor complexity: HSN, Simple MLP (no history), MLP+EMA, MLP+frame-stack (raw stack), GRU, LSTM, Transformer. HSN’s entire distribution lies above almost all medians of all six baselines in both environments.

Architecture	Exp. ( $\beta{=}1.5$ )	Mix-of-exp. (3-mode)
HSN, direct (no OLS)	$0.20_{{\pm.01}}$	$0.53_{{\pm.02}}$
HSN + OLS	$\mathbf{<10^{-6}}$	$\mathbf{<10^{-6}}$
Simple MLP + OLS	$0.72_{{\pm.03}}$	$1.09_{{\pm.03}}$
MLP+EMA + OLS	$0.38_{{\pm.01}}$	$0.60_{{\pm.02}}$
MLP+frame-stack + OLS	$0.26_{{\pm.04}}$	$0.43_{{\pm.03}}$
GRU + OLS	$1.00_{{\pm.35}}$	$1.39_{{\pm.45}}$

Table 2. Intensity MSE on a

400

-sample held-out set under the linear-probe protocol (

n=0.95

, single-asset, 10 seeds; SEM in subscript). The first row (“HSN, direct”) reports HSN’s zero-fitted-parameter direct estimate

\hat{\lambda}^{c}=\lambda_{0}^{c}+\sum_{j}h^{c}_{j}

evaluated on the held-out samples with no OLS fit; every other row uses the same OLS linear probe fit from that architecture’s feature vector to the ground-truth

(\lambda^{+},\lambda^{-})

on 1,600 training samples (indicated by the “

+

OLS” suffix).

Panel of Figure 2 — Figure 2. HSN $p$ -sweep on three single-asset kernels ( $n=0.85$ , $T=10$ ; 20 seeds per cell). (a) The $L^{2}$ kernel-approximation error; (b) the intensity-representation error. Both quantities decrease with $p$ and track each other across the three kernel classes.

Explore similar work

Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning
Sep 19, 2026Hao Yang, Zhenguo XuFinance Algorithmic Trading
North China Institute of Computer System Engineering, Beijing, China · University of Science and Technology of China, Hefei, China
Zero-shot adaptation to order book dynamics
May 20, 2026Arip AsadulaevSequential Decision Making Stochastic Optimal Control
Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions
Aug 19, 2026Tomasz R. Bielecki, Thibaut Mastrolia, Haoze YanStochastic Optimal Control Continuous-Time RL
Hinsdale, Illinois, USA · UC Berkeley, Department of Industrial Engineering and Operations Research, USA

Sep 19, 2026J/K move · Enter open · S save

Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning

Hao Yang, Zhenguo Xu

North China Institute of Computer System Engineering, Beijing, China · University of Science and Technology of China, Hefei, China

Finance Algorithmic Trading Reinforcement Learning Robust RL

Open paper AskPDF arXiv

Market-making strategies in real limit order book markets face substantial model uncertainty and regime-shift risk. Existing adversarial reinforcement learning approaches improve robustness by formulating the Avellaneda--Stoikov market-making problem as a zero-sum game between a market maker and an environmental adversary. However, these approaches typically rely on Poisson order arrivals and neglect trade-induced price impact, limiting their ability to capture important high-frequency market microstructure effects such as clustered order flow, self-excitation, and post-trade price feedback. We extend adversarial reinforcement learning for market making to a more complex environment with Hawkes self-exciting order arrivals and trade-induced price impact. To mitigate the increased non-stationarity introduced by the expanded regime space, we incorporate an LSTM module that explicitly models the temporal structure of recent observations. We further characterize the equilibrium properties of the proposed framework through both game-theoretic analysis and numerical experiments, and introduce a robustness evaluation protocol focused on improvements in the left tail of the return distribution. Experimental results across a range of market regimes show that the proposed method achieves improved left-tail performance in most complex microstructure environments. In particular, the gains are pronounced in regimes with strong Hawkes excitation and low-to-moderate price impact. Bootstrap tests provide no evidence that these improvements are obtained through a stronger terminal directional inventory bias. These results suggest that combining adversarial training with temporal state representation can improve the robustness of reinforcement-learning-based market-making strategies under order-flow self-excitation, price impact, and regime uncertainty.

Abstract

Explore similar work

Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning

Zero-shot adaptation to order book dynamics

Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions