Adversarial Post-Training of Foundation Models for High-Frequency Trading
Authors: Alexandre Bismuth, Kang Li, Aramis Marti-Shahandeh, Valentin Mohl, Jakob Foerster, Anisoara Calinescu
Organizations: British Open-ended Learning and Discovery Lab (BOLD), Department of Computer Science, University of Oxford, Oxford, United Kingdom · Harvard University, Cambridge, MA, United States · British Open-ended Learning and Discovery Lab (BOLD), Department of Statistics, University of Oxford, Oxford, United Kingdom · British Open-ended Learning and Discovery Lab (BOLD), Department of Engineering Science, University of Oxford, Oxford, United Kingdom
While autoregressive foundation models trained on limit order book (LOB) data show state-of-the-art performance for market simulation, pre-training with cross-entropy neither penalizes the compounding of sampling errors over free-running rollouts nor targets desired market properties. We introduce a post-training method for LOB foundation models using a Wasserstein generative adversarial network (GAN) objective. A critic, trained online on rollouts through the true, non-differentiable matching engine, guides updates from evolution strategies over rank-4 LoRA perturbations of the pre-trained generator, under a KL trust region. For efficiency, we use EGGROLL (Evolution Guided GeneRal Optimisation via Low-rank Learning), a memory-efficient variant of evolution strategies. On a held-out panel, adversarial post-training improves LOB-Bench realism by 17.6% (WS-21) and 17.0% (Mean-L1), while selection-matched random-perturbation and compute-matched continued-pre-training controls yield no improvement. Post-training is thus a necessary stage that more pre-training does not replace. Per-position divergence measurements show that it slows error accumulation over 500-message rollouts from 3.1× to 2.3×. Adversarial post-training is also inexpensive, requiring under 50M training tokens and 12 GPU-hours per seed, three orders of magnitude below the 48B-token pre-training budget. Ablations over critic inputs, optimizers, and the population-rollout trade-off, together with an analysis of reward hacking, yield practical design guidelines for adversarial post-training.
Figures & tables
Figure 1. Market simulation (left): a model continues LOB messages through a non-differentiable exchange simulator. Adversarial post-training (center): a critic trained online scores generated rollouts against real message streams, and EGGROLL converts scores into rank-4 LoRA ( Hu et al., 2022 ) perturbations under a KL trust region. Evaluation (right): offline selection with LOB-Bench and held-out evaluation to mitigate reward hacking. Solid arrows carry data; dashed arrows carry learning signals. Three-panel diagram. Left: a model continues order-book messages through an exchange simulator. Center: a critic scores generated rollouts against real ones and EGGROLL turns the scores into low-rank weight updates. Right: checkpoints are selected with LOB-Bench and scored on held-out data.
Figure 2. The limit order book and its message stream from records in LOBSTER format. Bar length represents volume. A limit order book drawn as bid and ask volume bars at each price level. Example messages (a cancellation, an execution and a new limit order) point to the levels they change, and a legend lists the six fields of a LOBSTER record.
Period
Msgs/day (M)
Exec. (%)
Spread (bps)
Daily RV (%)
Mid ($)
2022–2025
4.41
1.45
1.18
1.44
157
2025 Q1
5.95
1.20
1.25
1.50
183
2025 Q2
6.87
1.55
1.20
1.80
165
2025 Q3
5.15
1.40
1.14
1.31
210
2025 Q4
8.03
0.88
1.48
1.56
286
Test
6.17
1.29
1.24
1.70
329
Table 1. Statistics for the GOOG stock.
Figure 3. Raw-message critic. Each 26-token message is decoded into 12 channels while absolute values and IDs are withheld as trivial tells. Channels are median-centered and scaled by 1.4826⋅MAD (median absolute deviation) to match a Gaussian distribution. Spectral-normalized TCN members, each behind its own fixed random projection, score the 500-message continuation and the generator receives the pessimistic ensemble score. Each tokenized message is decoded into twelve numeric channels, normalized, and passed to an ensemble of convolutional critics whose pessimistic score is sent to the generator.
Model
Δ WS-21 [95% CI]
Δ Mean L1 [95% CI]
Pre-Trained
—
—
Post-Trained
−17.6% [-19.7, -15.3]
−17.0% [-18.9, -15.8]
Selection-null
+5.3% [+2.5, +8.1]
+7.8% [+5.9, +9.6]
IsoFLOP control
+0.5% [-2.3, +3.4]
−0.4% [-2.2, +1.4]
Table 2. Sealed test at 4,096 held-out windows (28-day panel). Each Δ is paired against the frozen pre-trained generator on the same windows, with window-block bootstrap 95% CIs; the post-trained Δ is the mean over n=10 seeds.
Critic input
Δ WS-21 ↓
Δ Mean L1↓
Generator backbone
−21.4%[−11.5,−28.2]
−18.7%[−10.0,−23.7]
Microstructure descriptors
−8.5%[−2.4,−11.6]
−6.7%[+0.2,−13.3]
Raw messages
−17.5%[−15.7,−19.7]
−18.9%[−17.2,−21.1]
Table 3. Critic-input ablation. Single-program sealed test at 2,048 held-out windows (28-day panel), all rows sharing one anchor; Δ = mean paired change vs. that anchor across three seeds, with the per-seed range in brackets.
Figure 4. KL(true∥gen)/H(true) accumulated since each curve’s first 25-message bin. Post-training accumulates 26% less excess divergence than the anchor, the selection null 27% more, continued pre-training the same. Shaded: 95% day-clustered bootstrap bands. Line plot of accumulated divergence against position in a 500-message rollout. The pre-trained model and continued pre-training rise together, the selection null rises fastest, and post-training rises the slowest.
Figure 5. Per-feature error shift at test-time, averaged over the 10 post-training seeds. The saturating power-law fit ( R2=0.86 , Spearman ρ=−0.76 ) shows the largest anchor errors improving most; features near the evaluation noise floor (left) swing widely in relative terms, and spread (annotated) is the main outlier. Scatter plot of relative error change after post-training against the pre-trained model's error for 21 features, with a decreasing power-law fit. Features with larger initial errors improve more; spread is marked as an outlier.
Horizon (messages)
Δ Directional accuracy [95% CI]
100
+0.011 [-0.008, +0.031]
250
+0.012 [-0.007, +0.028]
500
−0.004 [-0.024, +0.016]
1,000
+0.009 [-0.010, +0.027]
Table 4. Downstream mid-price direction at 2,048 held-out windows (28-day panel). Δ is the mean change in three-class directional accuracy across n=10 seeds relative to the frozen pre-trained anchor, with paired day-clustered bootstrap 95% CIs. Accuracy is preserved at every horizon: no interval separates from zero.