Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books
Authors: Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh, George Nigmatulin, Kang Li, Stefan Zohren, Jakob Foerster, Mihai Cucuringu, Anisoara Calinescu
Organizations: British Open-ended Learning and Discovery Lab (BOLD), Department of Computer Science, University of Oxford, Oxford, United Kingdom · Harvard University, Cambridge, MA, United States · British Open-ended Learning and Discovery Lab (BOLD), Department of Engineering Science, University of Oxford, Oxford, United Kingdom · British Open-ended Learning and Discovery Lab (BOLD), Department of Statistics, University of Oxford, Oxford, United Kingdom · Department of Mathematics, University of California, Los Angeles, Los Angeles, United States · Oxford-Man Institute of Quantitative Finance, Department of Statistics, University of Oxford, Oxford, United Kingdom
Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs). Yet, without understanding their inner workings, research directions are hard to narrow and practical use in real-world trading environments remains uncertain. This study therefore asks what these models actually learn about market microstructure. We find that, similarly to games where sequence models build an internal representation of the board, an 8.2M-parameter Mamba-3 state-space model trained autoregressively on the tokenized message stream of eight large-cap US equities learns an internal representation of the market and its dynamics. In particular, order-flow imbalance, a central driver of short-horizon price moves, is decodable at R2≈0.59, far above controls and baselines. This representation extends to unseen stocks, suggesting that these models learn general representations rather than ticker-specific patterns. The model also generalizes across metrics beyond order-flow imbalance: results hold across traded volume and volatility. Additionally, the representation is causally steerable: pushing the state along a feature's axis shifts the model's next-message beliefs as that feature implies, monotonically in dose, consistent with the emergence of a world model.
Figures & tables
Figure 1. Probing loci in a message-only Mamba-3 model. At each token, the state carries over the previous state scaled by αt and adds two writes: the current token’s, weighted by γt , and the previous token’s, weighted by βt . From each layer we extract the per-token residual stream and the recurrent state ht(ℓ) , which accumulates across all messages of the window. Both are read out by the same kernel ridge probe. Diagram of a Mamba-3 block showing where the residual stream and the recurrent state are probed.
Figure 2. Held-out R2 (layer chosen by nested CV) decoded from the recurrent state and the residual stream at W=500 , same windows and probing procedure for both. Whiskers: 95% day-block bootstrap intervals. OFI, traded volume and realized volatility are decodable almost only from the recurrent state, while the touch is partly readable from both. Bar chart: market statistics decode from the recurrent state but not from the residual stream.
Target
Trained
fwd-CV
Random
Book
OFI L1 W=500
0.59
0.53
0.01
0.12
OFI L1 W=100
0.49
0.44
0.03
0.12
OFI L1 W=10
0.29
0.26
0.07
0.02
OFI top-5 W=500
0.61
0.51
0.01
0.08
Table 1. Held-out OFI R2 (layer chosen by nested CV) from the trained SSM state, compared with a random-weights twin and the best readout, chosen by nested CV, of the full book snapshot at the observation boundary (Book). We include forward-chaining cross-validation for robustness. Table of held-out R-squared for order-flow imbalance at lookback windows of 500, 100 and 10 messages, comparing the trained state under k-fold and forward-chaining cross-validation with the random-weights twin and the book-snapshot baseline.
Target ( W=500 )
Trained
fwd-CV
Random
Book
Traded volume
0.50
0.44
0.03
0.11
Realized volatility
0.51
0.50
0.15
0.23
Log-vol L10 book depth
0.38
0.30
0.09
0.64
Table 2. Held-out R2 from the SSM state (layer chosen by nested CV) vs. the random-weights twin and the best book-snapshot baseline, chosen by nested CV among all three snapshot sets (Book), including forward-chaining CV. For depth, Book is the stale window-start snapshot, since the boundary snapshot contains the target itself. Table of held-out R-squared for traded volume, realized volatility and book depth over 500-message windows, comparing the trained state under k-fold and forward-chaining cross-validation with the random-weights twin and the book-snapshot baseline.
Figure 3. Per-layer decodability for (a) OFI L1 ( W=500 ) and (b) touch (best-ask offset). We test four probes of increasing capacity: linear, quadratic, RBF kernel ridge, and a one-hidden-layer MLP on the top 384 principal components of the state. All four probes show the same pattern, so the mid-stack dip is not a non-linear re-encoding. Per-layer R-squared curves for OFI and touch under linear and non-linear probes.
Figure 4. Share of the recurrent state (blue; all layers) and of the decoded OFI signal (green; best layer) accounted for by the L most recent messages, for the trained model and its random-weights twin. Half of the raw state is the newest message alone; half of the OFI signal takes the last 56 messages. Cumulative share of the state and of the decoded OFI signal against the number of recent messages.
Figure 5. Model’s beliefs about the next message after pushing each feature’s state axis by a dose α : (a) order flow, (b) traded volume, and (c) realized volatility. Grey: mean and range of 30 norm-matched axes built from shuffled labels. Orange: a norm-matched random axis. Perplexity measures the effective width in one-tick levels of the next-price forecast. Three side-by-side panels, one per feature, showing the model's next-message belief against steering dose: the trained axis rises monotonically, the shuffled-label null stays near the unpushed value with a band that widens with dose, and the random axis is inert.
Figure 6. Three-class (down/flat/up) mid-price direction from a linear decoding of the recurrent state ( W=500 ) for different forward horizons H : (a) accuracy, (b) macro-F1. Grey dashed: book-snapshot baseline classifier, with the best feature set and kernel chosen by nested CV. Shaded bands: 95% day-block bootstrap CIs. Directional accuracy and macro-F1 against forecast horizon.
Figure 7. Market statistics decodability and LOB-Bench scores at different parts of training (0% = random-weights twin). Touch is learned early, OFI keeps improving over the whole run, and LOB-Bench plateaus after about a third of training. Decodability and LOB-Bench scores across training checkpoints.
Figure 8. Held-out R2 (layer chosen by nested CV) of probes on the frozen SSM state for ten stocks, W=500 . Black ticks represent the random-weights twin and the dashed divider separates in-training from out-of-distribution (OOD) stocks. (a) Touch decodes on all ten stocks, including the OOD stocks, while (b) OFI R2 varies widely. Per-stock R-squared for touch and OFI, including out-of-distribution stocks, with the random-weights twin marked on every bar.
Figure 9. (a) OFI L1 R2 versus the top-1% windows’ share s of target variance. Tail prevalence explains most of the variation across training stocks (fitted line; fit R2=0.94 , Spearman rs=−0.93 , p=0.002 ), with the two OOD stocks about 0.15 below. (b) Raw vs. winsorized-1% vs. rank-transform R2 for a representative subset: both tail-robust metrics lift every stock above R2=0.4 and narrow the spread across stocks. OFI R-squared against the tail share of target variance, and robust-target comparison.
Autoregressive transformers trained on limit order book (LOB) message streams are increasingly proposed as foundation models for market microstructure. Because the generative process of real markets is unknown, these models can be judged only by the realism of their output, leaving open what they actually learn. We make the question answerable by inverting the setup: we train GPT-style models on order flow from simulators whose latent parameters are known, so that the Bayes-optimal predictor, the generating state, and the exact data entropy serve as references. Measured against them, next-token pretraining on order flow implements amortized Bayesian inference over market regimes. The learned predictive matches the exact Bayes optimum to within a millinat per token and approaches the oracle floor even on a nonstationary family whose latent state drifts within an episode, a regime that requires filtering. The latent regime is linearly decodable from frozen activations, well above baselines, so a frozen-model read-out calibrates simulator parameters more accurately than windowed maximum likelihood at a fraction of the cost. In our experiments these capabilities are not produced by scale: capacity and token volume saturate early, and it is the diversity of training regimes, not the token count, that determines generalization to unseen markets. On real NASDAQ data, zero-shot transfer is set by how well the simulator prior covers real flow: an enriched mixture nearly closes the gap to an in-domain baseline, and pretraining beats from-scratch training at every budget across held-out tickers. Known ground truth turns average next-token loss, which is uninformative near the entropy floor, into an exact comparison against the Bayes-optimal and oracle predictors.
Modeling the dynamics of financial Limit Order Books (LOB) at the message level is challenging due to irregular event timing, rapid regime shifts, and the reactions of high-frequency traders to visible order flow. Previous LOB models require cumbersome data representations and lack adaptability outside their original tasks, leading us to introduce LOBERT, a general-purpose encoder-only foundation model for LOB data suitable for downstream fine-tuning. LOBERT adapts the original BERT architecture for LOB data by using a novel tokenization scheme that treats complete multi-dimensional messages as single tokens while retaining continuous representations of price, volume, and time. With these methods, LOBERT achieves leading performance in tasks such as predicting mid-price movements and next messages, while reducing the required context length compared to previous methods.
Eljas Linna, Kestutis Baltakys, Alexandros Iosifidis +1
A large language model (LLM) trained on synthetic limit order book (LOB) data achieves near perfect scores in generating valid sequences of LOB events. However, the LLM's implicit world model fails to learn the state of the LOB. This deficiency leads to biased estimates and spurious predictability in using the LLM to forecast future LOB events. Our analysis uses novel tests of an LLM's world model, extending prior work from deterministic settings to the stochastic dynamics needed for the LOB.