What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books

Fernando Acero

cs.LGOct 14, 2026

What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books

Authors: Fernando Acero

Organizations: J.P. Morgan QTR AI Research, United Kingdom

Abstract

Autoregressive transformers trained on limit order book (LOB) message streams are increasingly proposed as foundation models for market microstructure. Because the generative process of real markets is unknown, these models can be judged only by the realism of their output, leaving open what they actually learn. We make the question answerable by inverting the setup: we train GPT-style models on order flow from simulators whose latent parameters are known, so that the Bayes-optimal predictor, the generating state, and the exact data entropy serve as references. Measured against them, next-token pretraining on order flow implements amortized Bayesian inference over market regimes. The learned predictive matches the exact Bayes optimum to within a millinat per token and approaches the oracle floor even on a nonstationary family whose latent state drifts within an episode, a regime that requires filtering. The latent regime is linearly decodable from frozen activations, well above baselines, so a frozen-model read-out calibrates simulator parameters more accurately than windowed maximum likelihood at a fraction of the cost. In our experiments these capabilities are not produced by scale: capacity and token volume saturate early, and it is the diversity of training regimes, not the token count, that determines generalization to unseen markets. On real NASDAQ data, zero-shot transfer is set by how well the simulator prior covers real flow: an enriched mixture nearly closes the gap to an in-domain baseline, and pretraining beats from-scratch training at every budget across held-out tickers. Known ground truth turns average next-token loss, which is uninformative near the entropy floor, into an exact comparison against the Bayes-optimal and oracle predictors.

Figures & tables

Panel of Figure 1 — Figure 1. Study design. Simulators with known $\boldsymbol{\theta}$ generate message streams through a matching engine. GPT-style models train on the tokens. Each question has an exact reference: the predictive vs. exact Bayes or oracle (RQ1), probes vs. the true $\boldsymbol{\theta}$ (RQ2), and scaling fits vs. the true entropy floor (RQ3).

family	parameter	prior	role
all	$\Lambda$ (rate, s -1 )	LogU $(0.5,50)$	activity
F1 , F2	$g$ (LO mass)	Beta $(137,103)$	book stability
	bid/ask split	Dir $(4,4)$	side asymmetry
	MO/CX split	Dir $(2,2,3,3)$	flow mix
	$p_{\mathrm{geo}}$ (offsets)	$U(0.15,0.6)$	book depth
	$\bar{s}$ (sizes)	LogU $(1,60)$	order sizes

Table 1. Simulator families (latent parameters and priors) and model sizes. All families share the engine, tokenizer, and episode structure (2,000 burn-in + 16,384 recorded messages). LogU = log-uniform; Dir = Dirichlet.

Figure 2. Statistical properties of the families (40 episodes each; 150 for F3 ). (a) Inter-arrival and (b) one-second activity autocorrelation: F2 clusters, F1 does not. (c) F3 has heavy return tails. (d) Market-maker weight tightens spreads (1.0 vs. 17.4 ticks across terciles). (e) Momentum weight raises variance ratios (trend persistence; 1 = random walk). (f) Book depth in one episode.

Figure 3. Mean prediction loss vs. window position

n

for S2 models (400 held-out episodes per family; 9-point smoothing). Solid: trained model; dashed: exact Bayes ( F1 ); dotted: oracle given

\boldsymbol{\theta}

. The model matches exact Bayes on F1 (a) and reaches the oracle floors elsewhere, including the moving latent state of F4 (d), which requires filtering. The model in (d) is the four-family mixture.

family	5-gram	transformer (S2)	irreducible floor
F1	1.575	1.3379	1.3365 (exact Bayes-window)
F2	1.568	1.3359	1.3271 (oracle)
F3	1.439	1.2718	1.2552 (oracle)

Table 2. Validation loss (nats per token) of a backoff 5-gram model over the token stream, the trained transformer (size S2, Table 1 ), and the exactly known irreducible floor of each family. The 5-gram captures short-range structure, and its gap to the transformer measures the value of long-context inference over the latent regime. F4 is omitted: its nonstationary floor is not a single per-token constant.

target	trained	random-init	summary stats	MLE
$a$ ( F2 )	$\mathbf{0.83}\pm.02$	$0.66\pm.02$	$0.73\pm.02$	$-0.18$
$\log\beta$ ( F2 )	$\mathbf{0.70}\pm.03$	$0.07\pm.04$	$0.42\pm.03$	$-4.89$
$w_{mom}$ ( F3 )	$\mathbf{0.92}\pm.01$	$0.42\pm.05$	$0.52\pm.03$	–
$w_{mm}$ ( F3 )	$\mathbf{0.97}\pm.01$	$0.81\pm.06$	$0.92\pm.01$	–
$\log\Lambda$ ( F2 )	0.98	0.93	0.97	–
$p_{geo}$ ( F2 )	0.95	0.95	0.87	–

Table 3. Test

R^{2}

of linear probes from the trained S2 model’s activations to latent generator parameters at

m{=}512

messages of context, against falsification baselines. Dynamical parameters: mean

\pm

sd over 5 episode-grouped CV folds at the best layer. Static parameters (below the rule) are recovered by all methods and shown from a single split. MLE = windowed Hawkes maximum likelihood (

R^{2}

; negative = worse than predicting the mean).

Figure 4. Probe $R^{2}$ by layer (0 = embeddings) and context length $m$ , for the trained S2 model (top) and an untrained network (bottom); yellow = accurate recovery. Composition weights peak at the deepest layer, and the advantage of training is largest at short contexts.

	FM probe	NPE-MLP	Hawkes MLE
RMSE $a$	0.087	0.114	0.236
RMSE $\log\beta$	0.556	0.727	2.490
sec / window	0.02	$\sim$ 0.02	5.5

Table 4. Calibration of the F2 parameters

(a,\log\beta)

from single 512-message windows: root-mean-square recovery error on held-out windows and wall-clock time per window (same GPU). FM probe: linear read-out of the frozen pretrained S2 model. NPE-MLP: a multilayer perceptron trained on the 22 summary statistics for this task. Hawkes MLE: windowed maximum likelihood.

Figure 5. Scaling with a known floor. (a) F2 , (b) F1 : loss vs. training tokens for the S-series sizes S0–S3 (points) with the joint Chinchilla-style fit (curves); horizontal lines mark the fitted

E

and the known floors. (c) Excess loss over the floor at

D{=}400

M vs. parameter count, now including the micro sizes m0–m2 that expose the low-capacity regime: capacity binds only below roughly

10^{5}

parameters.

Figure 6. Diversity ablation: S1 models, fixed 400M-token budget,

K

distinct regimes in training (two seeds; bands span seeds). (a) Loss on unseen regimes and under a shifted prior. (b) Best-layer probe

R^{2}

m{=}512

: decodability of the harder parameter is created by diversity.

model $\backslash$ eval	F1	F2	F3
F1	1.337	1.367	2.086
F2	1.339	1.338	2.082
F3	3.164	3.271	1.272
MIX	1.337	1.339	1.277

Table 5. Cross-family zero-shot val loss (nats/token): every model evaluated on every family. The mixture generalist matches each specialist on its home family within millinats.

Zero-shot per field (mean over 5 tickers)
	type	price	size	$\Delta t$
mixture ( F1 – F3 )	1.53	1.30	1.60	3.07
+ F4 enrichment	1.48	1.29	0.91	2.81
unigram (in-domain)	1.50	1.59	0.90	1.95
Fine-tune vs. scratch, mean over 3 held-out tickers
	5% slice	20%	80%	zero-shot

Table 6. Real-data transfer (NASDAQ LOBSTER, 2012-06-21; nats/token). Top: zero-shot per-field loss averaged over five tickers for the original mixture, the F4 -enriched mixture, and a unigram fit on the same day. Bold marks the better transfer model per field (the unigram is an in-domain reference, not a transfer model). Bottom: fine-tuning the enriched model on an AMZN+MSFT slice vs. from-scratch training on it, averaged over three fully held-out tickers (INTC, AAPL, GOOG).

Figure 7. Agent-composition probe read-outs on real NASDAQ streams. Each point estimates one agent weight on one 512-message window.

Limit Order Book Modeling

See all 20 papers

Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books
Oct 20, 2026Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh +6Limit Order Book Modeling State Representation Learning
British Open-ended Learning and Discovery Lab (BOLD), Department of Computer Science, University of Oxford, Oxford, United Kingdom · Harvard University, Cambridge, MA, United States · British Open-ended Learning and Discovery Lab (BOLD), Department of Engineering Science, University of Oxford, Oxford, United Kingdom +3
LOBERT: Generative AI Foundation Model for Limit Order Book Messages
Nov 16, 2025Eljas Linna, Kestutis Baltakys, Alexandros Iosifidis +1Limit Order Book Forecasting Quantitative Finance
Data Science Research Centre Tampere University
LOB-Impact Benchmark: Emergent Market Impact in Generative Limit Order Book Models
Oct 19, 2026George Nigmatulin, Valentin Mohl, Sascha Frey +6Limit Order Book Modeling Generative Modeling
British Open-ended Learning and Discovery Lab (BOLD), Department of Engineering Science, University of Oxford, Oxford, United Kingdom · British Open-ended Learning and Discovery Lab (BOLD), Department of Computer Science, University of Oxford, Oxford, United Kingdom · British Open-ended Learning and Discovery Lab (BOLD), Department of Statistics, University of Oxford, Oxford, United Kingdom +2

Oct 20, 2026J/K move · Enter open · S save

Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books

Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh, George Nigmatulin+5

British Open-ended Learning and Discovery Lab (BOLD), Department of Computer Science, University of Oxford, Oxford, United Kingdom · Harvard University, Cambridge, MA, United States · British Open-ended Learning and Discovery Lab (BOLD), Department of Engineering Science, University of Oxford, Oxford, United Kingdom+3

Limit Order Book Modeling State Representation Learning Time-Series Foundation Models Mechanistic Interpretability State Space Models Quantitative Finance

Open paper AskPDF arXiv

Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs). Yet, without understanding their inner workings, research directions are hard to narrow and practical use in real-world trading environments remains uncertain. This study therefore asks what these models actually learn about market microstructure. We find that, similarly to games where sequence models build an internal representation of the board, an 8.2M-parameter Mamba-3 state-space model trained autoregressively on the tokenized message stream of eight large-cap US equities learns an internal representation of the market and its dynamics. In particular, order-flow imbalance, a central driver of short-horizon price moves, is decodable at

R^2 \approx 0.59

, far above controls and baselines. This representation extends to unseen stocks, suggesting that these models learn general representations rather than ticker-specific patterns. The model also generalizes across metrics beyond order-flow imbalance: results hold across traded volume and volatility. Additionally, the representation is causally steerable: pushing the state along a feature's axis shifts the model's next-message beliefs as that feature implies, monotonically in dose, consistent with the emergence of a world model.

Abstract

Limit Order Book Modeling

Explore similar work

Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books

LOBERT: Generative AI Foundation Model for Limit Order Book Messages

LOB-Impact Benchmark: Emergent Market Impact in Generative Limit Order Book Models