Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books

Skander Lejmi

stat.MLOct 19, 2026

Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books

Authors: Skander Lejmi

Organizations: Department of Statistics, London School of Economics and Political Science

Abstract

A market maker provides liquidity by quoting prices to buy and sell an asset, earning the spread while managing inventory risk. We study market makers trained with multi-agent reinforcement learning on prediction markets, specifically Polymarket, whose thin order books pose distinct challenges. Three challenges underpin this research: adverse selection may cause a policy trained on real data to stop quoting; the consequences of a quote can arrive many steps after the action; and prediction-market L2 data does not identify individual orders needed for simulation. We address these challenges by reconstructing order-level LOBSTER-format data from Polymarket and training agents in JaxMARL-HFT on 417 real markets. We enrich the baseline market maker's observation space with the book and recent flow, redesign its actions and inventory controls, and propose a cooperative multi-agent decomposition into a quoter and a risk manager trained using recurrent Independent Proximal Policy Optimisation (IPPO) with RNN policies. We investigate reward shaping, particularly reward blending and post-fill markout, to mitigate credit-assignment difficulties between the agents. Our redesigned single-agent market maker quotes more frequently and incurs smaller held-out losses than the retrained JaxMARL-HFT baseline, although the contributions of jointly changed observations, actions and risk constraints are not separately identified. Decomposition alone performs poorly; careful reward design substantially improves the desk relative to the unshaped split and yields our best result. Models are evaluated on realised end-of-episode portfolio value on held-out markets. None establishes positive average portfolio value in periodic held-out evaluation, and each configuration was trained with one seed. The environment, data and protocol lay a foundation for makers that quote many linked markets at once.

Figures & tables

quantity	value
recorded messages per step	400
episode length	250 steps (100,000 messages)
book depth	10 levels a side loaded, 5 observed
parallel environments	64
transitions per update	$64\times 250$ , in 2 minibatches, 4 epochs
configured budget	3.2 million environment steps (200 updates)

Table 1: Quantities fixed across every configuration.

component	setting
quote table	the seventeen-row wide table of section 3.6.1 , plus stand-aside
observation	the book-and-flow vector of section 3.5
auxiliary head	the mid-price forecast head of section 4.1.2 , coefficient $0.1$
inventory cap	4 units
network	256 recurrent units, 256 per dense layer

Table 2: The stack shared by our models.

setting	value
optimiser	Adam, learning rate $2.5\times 10^{-4}$ , constant
discount $\gamma$	$\approx 1$ (market making, risk manager); $0.999$ (execution)
GAE trace $\lambda$	$0.97$ (market making, risk manager); $0.9$ (execution)
clipping $\epsilon$	$0.2$
entropy coefficient	$0.01$
value coefficient	$0.5$

Table 3: Settings common to every model. Where a value differs by agent type, both are given (i.e. market maker, execution). The quoter and risk manager take the market maker’s value unless otherwise specified.

model	horizons (steps)	forecast to actor
D_SPLIT , D_BLEND , D_MARK	1, 5, 10	no
WIDE_SPOONER	1, 5, 10	yes
D_BL_MARK_CURR	1, 10, 60	yes

Table 4: Forecast-head settings.

model	blend	markout
D_SPLIT	$0$	–
D_BLEND	$0.3$	–
D_MARK	$0$	$0.3$ / $5$
D_BL_MARK_CURR	$0.3$	$0.15$ / $5$ , ramped

Table 5: Desk alignment terms. Markout is given as a coefficient (that is applied to the markout term in the quoter’s reward), where it is valid, it applies with a step horizon of

5

. Ramped means the coefficient is scaled in linearly over the first

30\%

of the run.

configuration	PV	spread	volume	no-trade	$\|$ inv $\|$
	(ticks)		(units/step)	(%)	(units)
PAPER_BASE	$-46.1$	$19.4$	$0.056$	$19.2$	$1.108$
WIDE_SPOONER	$-9.6$	$3.1$	$0.032$	$4.4$	$0.366$

Table 6: Non-desk configurations. Portfolio value is the held-out end-of-episode value in ticks, averaged over the last five evaluations; spread is its standard deviation over the last ten, so a smaller figure is a run that has settled. No-trade is the share of training steps on which the policy selects the no-trade action. One seed per configuration.

configuration	PV	spread	volume	no-trade	$\|$ inv $\|$
	(ticks)		(units/step)	(%)	(units)
D_SPLIT	$-72.3$	$17.6$	$0.112$	$1.6$	$0.241$
D_BLEND	$-41.0$	$12.3$	$0.068$	$4.7$	$0.172$
D_MARK	$-5.9$	$3.5$	$0.032$	$20.4$	$0.082$
D_BL_MARK_CURR	$-4.3$	$1.7$	$0.022$	$19.7$	$0.060$

Table 7: Desk configurations, same columns. The middle two rows each add one term to D_SPLIT ; the last combines both with the loss-stop curriculum.

Algorithmic Trading

Algorithmic trading uses computational models to generate signals, allocate portfolios, and execute orders across financial markets.

See all 149 papers

Apr 4, 2026

PolySwarm: A Multi-Agent Large Language Model Framework for Prediction Market Trading and Latency Arbitrage

This paper presents PolySwarm, a novel multi-agent large language model (LLM) framework designed for real-time prediction market trading and latency arbitrage on decentralized platforms such as Polymarket. PolySwarm deploys a swarm of 50 diverse LLM personas that concurrently evaluate binary outcome markets, aggregating individual probability estimates through confidence-weighted Bayesian combination of swarm consensus with market-implied probabilities, and applying quarter-Kelly position sizing for risk-controlled execution. The system incorporates an information-theoretic market analysis engine using Kullback-Leibler (KL) divergence and Jensen-Shannon (JS) divergence to detect cross-market inefficiencies and negation pair mispricings. A latency arbitrage module exploits stale Polymarket prices by deriving CEX-implied probabilities from a log-normal pricing model and executing trades within the human reaction-time window. We provide a full architectural description, implementation details, and evaluation methodology using Brier scores, calibration analysis, and log-loss metrics benchmarked against human superforecaster performance. We further discuss open challenges including hallucination in agent pools, computational cost at scale, regulatory exposure, and feedback-loop risk, and outline five priority directions for future research. Experimental results demonstrate that swarm aggregation consistently outperforms single-model baselines in probability calibration on Polymarket prediction tasks.

Rajat M. Barot, Arjun S. Borkhatariya

Department of Computer Science, State University of New York, Binghamton, Binghamton, NY, USA. · Department of Software Engineering, Arizona State University, Tempe, AZ, USA.

Algorithmic Trading Prediction Markets Multi-Agent LLMs Probabilistic Forecasting

Ask PDF

Sep 19, 2026

Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning

Market-making strategies in real limit order book markets face substantial model uncertainty and regime-shift risk. Existing adversarial reinforcement learning approaches improve robustness by formulating the Avellaneda--Stoikov market-making problem as a zero-sum game between a market maker and an environmental adversary. However, these approaches typically rely on Poisson order arrivals and neglect trade-induced price impact, limiting their ability to capture important high-frequency market microstructure effects such as clustered order flow, self-excitation, and post-trade price feedback. We extend adversarial reinforcement learning for market making to a more complex environment with Hawkes self-exciting order arrivals and trade-induced price impact. To mitigate the increased non-stationarity introduced by the expanded regime space, we incorporate an LSTM module that explicitly models the temporal structure of recent observations. We further characterize the equilibrium properties of the proposed framework through both game-theoretic analysis and numerical experiments, and introduce a robustness evaluation protocol focused on improvements in the left tail of the return distribution. Experimental results across a range of market regimes show that the proposed method achieves improved left-tail performance in most complex microstructure environments. In particular, the gains are pronounced in regimes with strong Hawkes excitation and low-to-moderate price impact. Bootstrap tests provide no evidence that these improvements are obtained through a stronger terminal directional inventory bias. These results suggest that combining adversarial training with temporal state representation can improve the robustness of reinforcement-learning-based market-making strategies under order-flow self-excitation, price impact, and regime uncertainty.

Hao Yang, Zhenguo Xu

North China Institute of Computer System Engineering, Beijing, China · University of Science and Technology of China, Hefei, China

Adversarial MARL Hawkes Processes Finance Algorithmic Trading Reinforcement Learning Robust RL

Ask PDF

Sep 29, 2026

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets

Power markets are a natural testbed for multi-agent reinforcement learning (MARL), where multiple self-interested participants repeatedly submit bids. A market-clearing mechanism then determines dispatch and prices subject to power grid constraints and market settlement rules. However, existing MARL environments typically focus on a single market setting, implement simplified clearing mechanisms, or rely on CPU-based optimization solvers that slow large-scale training and limit the systematic study of bidding strategies and market behavior. We introduce PowerMarketJax, a benchmark suite for MARL across five power markets: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer double auctions, and local flexibility. Each environment implements its own clearing, pricing, and settlement rules while providing a common framework for learning and evaluation. We find that learned bidding behavior depends strongly on the market design: independent learners can miss better strategies when gains require many agents to change together, when more profitable strategies lie beyond a region of lower profit, or when profits disappear as more agents adopt the same strategy. PowerMarketJax implements both market simulation and policy training in JAX, allowing the entire pipeline to run on the GPU with 1,024 X 1,200 parallelisms across both environments and market participants, achieving up to 33X speedup over CPU-based baselines. Our open-source benchmark is available at: https://github.com/powermarketjax/PowerMarketJax.

Zhanhua Pan, Xin Qin, Xiao Liu +3

Nanyang Technological University, Singapore. · Cornell University, USA. · University of Bristol, UK.

RL Benchmarks Multi-Agent Reinforcement Learning

Ask PDF

Abstract

Algorithmic Trading

Explore similar work

PolySwarm: A Multi-Agent Large Language Model Framework for Prediction Market Trading and Latency Arbitrage

Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets