Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books
Authors: Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh, George Nigmatulin, Kang Li, Stefan Zohren, Jakob Foerster, Mihai Cucuringu, Anisoara Calinescu
Organizations: British Open-ended Learning and Discovery Lab (BOLD), Department of Computer Science, University of Oxford, Oxford, United Kingdom · Harvard University, Cambridge, MA, United States · British Open-ended Learning and Discovery Lab (BOLD), Department of Engineering Science, University of Oxford, Oxford, United Kingdom · British Open-ended Learning and Discovery Lab (BOLD), Department of Statistics, University of Oxford, Oxford, United Kingdom · Department of Mathematics, University of California, Los Angeles, Los Angeles, United States · Oxford-Man Institute of Quantitative Finance, Department of Statistics, University of Oxford, Oxford, United Kingdom
Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs). Yet, without understanding their inner workings, research directions are hard to narrow and practical use in real-world trading environments remains uncertain. This study therefore asks what these models actually learn about market microstructure. We find that, similarly to games where sequence models build an internal representation of the board, an 8.2M-parameter Mamba-3 state-space model trained autoregressively on the tokenized message stream of eight large-cap US equities learns an internal representation of the market and its dynamics. In particular, order-flow imbalance, a central driver of short-horizon price moves, is decodable at R2≈0.59, far above controls and baselines. This representation extends to unseen stocks, suggesting that these models learn general representations rather than ticker-specific patterns. The model also generalizes across metrics beyond order-flow imbalance: results hold across traded volume and volatility. Additionally, the representation is causally steerable: pushing the state along a feature's axis shifts the model's next-message beliefs as that feature implies, monotonically in dose, consistent with the emergence of a world model.
Figures & tables
Figure 1. Probing loci in a message-only Mamba-3 model. At each token, the state carries over the previous state scaled by αt and adds two writes: the current token’s, weighted by γt , and the previous token’s, weighted by βt . From each layer we extract the per-token residual stream and the recurrent state ht(ℓ) , which accumulates across all messages of the window. Both are read out by the same kernel ridge probe. Diagram of a Mamba-3 block showing where the residual stream and the recurrent state are probed.
Figure 2. Held-out R2 (layer chosen by nested CV) decoded from the recurrent state and the residual stream at W=500 , same windows and probing procedure for both. Whiskers: 95% day-block bootstrap intervals. OFI, traded volume and realized volatility are decodable almost only from the recurrent state, while the touch is partly readable from both. Bar chart: market statistics decode from the recurrent state but not from the residual stream.
Target
Trained
fwd-CV
Random
Book
OFI L1 W=500
0.59
0.53
0.01
0.12
OFI L1 W=100
0.49
0.44
0.03
0.12
OFI L1 W=10
0.29
0.26
0.07
0.02
OFI top-5 W=500
0.61
0.51
0.01
0.08
Table 1. Held-out OFI R2 (layer chosen by nested CV) from the trained SSM state, compared with a random-weights twin and the best readout, chosen by nested CV, of the full book snapshot at the observation boundary (Book). We include forward-chaining cross-validation for robustness. Table of held-out R-squared for order-flow imbalance at lookback windows of 500, 100 and 10 messages, comparing the trained state under k-fold and forward-chaining cross-validation with the random-weights twin and the book-snapshot baseline.
Target ( W=500 )
Trained
fwd-CV
Random
Book
Traded volume
0.50
0.44
0.03
0.11
Realized volatility
0.51
0.50
0.15
0.23
Log-vol L10 book depth
0.38
0.30
0.09
0.64
Table 2. Held-out R2 from the SSM state (layer chosen by nested CV) vs. the random-weights twin and the best book-snapshot baseline, chosen by nested CV among all three snapshot sets (Book), including forward-chaining CV. For depth, Book is the stale window-start snapshot, since the boundary snapshot contains the target itself. Table of held-out R-squared for traded volume, realized volatility and book depth over 500-message windows, comparing the trained state under k-fold and forward-chaining cross-validation with the random-weights twin and the book-snapshot baseline.
Figure 3. Per-layer decodability for (a) OFI L1 ( W=500 ) and (b) touch (best-ask offset). We test four probes of increasing capacity: linear, quadratic, RBF kernel ridge, and a one-hidden-layer MLP on the top 384 principal components of the state. All four probes show the same pattern, so the mid-stack dip is not a non-linear re-encoding. Per-layer R-squared curves for OFI and touch under linear and non-linear probes.
Figure 4. Share of the recurrent state (blue; all layers) and of the decoded OFI signal (green; best layer) accounted for by the L most recent messages, for the trained model and its random-weights twin. Half of the raw state is the newest message alone; half of the OFI signal takes the last 56 messages. Cumulative share of the state and of the decoded OFI signal against the number of recent messages.
Figure 5. Model’s beliefs about the next message after pushing each feature’s state axis by a dose α : (a) order flow, (b) traded volume, and (c) realized volatility. Grey: mean and range of 30 norm-matched axes built from shuffled labels. Orange: a norm-matched random axis. Perplexity measures the effective width in one-tick levels of the next-price forecast. Three side-by-side panels, one per feature, showing the model's next-message belief against steering dose: the trained axis rises monotonically, the shuffled-label null stays near the unpushed value with a band that widens with dose, and the random axis is inert.
Figure 6. Three-class (down/flat/up) mid-price direction from a linear decoding of the recurrent state ( W=500 ) for different forward horizons H : (a) accuracy, (b) macro-F1. Grey dashed: book-snapshot baseline classifier, with the best feature set and kernel chosen by nested CV. Shaded bands: 95% day-block bootstrap CIs. Directional accuracy and macro-F1 against forecast horizon.
Figure 7. Market statistics decodability and LOB-Bench scores at different parts of training (0% = random-weights twin). Touch is learned early, OFI keeps improving over the whole run, and LOB-Bench plateaus after about a third of training. Decodability and LOB-Bench scores across training checkpoints.
Figure 8. Held-out R2 (layer chosen by nested CV) of probes on the frozen SSM state for ten stocks, W=500 . Black ticks represent the random-weights twin and the dashed divider separates in-training from out-of-distribution (OOD) stocks. (a) Touch decodes on all ten stocks, including the OOD stocks, while (b) OFI R2 varies widely. Per-stock R-squared for touch and OFI, including out-of-distribution stocks, with the random-weights twin marked on every bar.
Figure 9. (a) OFI L1 R2 versus the top-1% windows’ share s of target variance. Tail prevalence explains most of the variation across training stocks (fitted line; fit R2=0.94 , Spearman rs=−0.93 , p=0.002 ), with the two OOD stocks about 0.15 below. (b) Raw vs. winsorized-1% vs. rank-transform R2 for a representative subset: both tail-robust metrics lift every stock above R2=0.4 and narrow the spread across stocks. OFI R-squared against the tail share of target variance, and robust-target comparison.