CrashGym: Measuring the Decision Value of Experimentation for LLM Agents in Counterfactual Market Crashes

Haocheng Lu; Haoxuan Cheng; Mingyang Chen; Minjun Zhu

cs.AIOct 21, 2026

CrashGym: Measuring the Decision Value of Experimentation for LLM Agents in Counterfactual Market Crashes

Authors: Haocheng Lu, Haoxuan Cheng, Mingyang Chen, Minjun Zhu

Organizations: Columbia University · Cornell University · Hithink RoyalFlush Information Network Co., Ltd. · New York University

Abstract

Language-model agents are increasingly asked to run experiments, yet benchmarks rarely measure whether those experiments improve the decisions they are meant to inform. We introduce CrashGym, a deterministic, replayable limit-order-book simulator in which an agent must choose between a deleveraging intervention and a liquidity backstop for a market about to suffer a shock-triggered crash. Before committing, the agent may query intervention experiments in a laboratory realization of the market. Its choice is then scored on an independently seeded target realization in which the simulator executes every feasible intervention, so regret relative to the best intervention is computed by directly simulating every alternative on the same target repeats, with no human or model-based judging. The simulator implements each amplification channel---persistent price impact, liquidity withdrawal, margin-driven liquidation, and an evolving strategy mix---as an explicit mechanism that can be switched on in stages. In a prospectively specified study on 24 newly sampled worlds, laboratory access reduces the mean regret of the one deployment with a complete record (Luna) from 8.878 to 3.785, a paired reduction of 5.092 utility units (nominal 95% world-bootstrap interval: 1.028--9.507), or 57.4% of the OBSERVE-ONLY mean. Prospective comparisons for two further deployments are incomplete; in a separate exploratory study, mean regret is lower with laboratory access for all four deployments evaluated. A laboratory whose strategy mix evolves endogenously yields no established advantage over one with a fixed mix. CrashGym thus measures the value of experimentation as avoided decision loss and lets simulator detail be judged by the decisions it supports.

Figures & tables

Panel of Figure 1 — Figure 1: Task overview. A world constructor $w$ is instantiated with two disjoint seed cohorts. (a) In the laboratory cohort the agent may run intervention experiments and receives public eight-repeat summaries. (b) Its committed action is scored on the other cohort, where the simulator executes both feasible interventions; regret is the utility gap to the better one. Target outcomes are never visible to the agent. Under Observe-only , the agent decides from the initial observation alone.

Participant	Rule per step	Active from
Market maker	40-level ladder per side at $\pm(2+k)$ ticks; $\bar{Q}$ lots/level at step 0, 400 during steps 1–7, $\bar{Q}$ from the shock on, thinned under stress (Eq. 1 )	all configurations
Noise traders	8 marketable orders of 4 lots, side drawn with probability $1/2$	all configurations
Leveraged trader	buys 160–240 lots at step 1; forced sales when below maintenance margin (Eq. 2 )	margin rule: leverage stage
Fundamental trader	4 lots per tick of $v-\text{mid}$ , marketable	leverage stage
Trend follower	$4e_{t}$ lots, $e_{t}=\operatorname{trunc}((3e_{t-1}+r_{t-1})/4)$ , marketable	leverage stage
Adaptive participant	after the shock, draws the Fundamental or Trend candidate from shares $(s_{\mathrm{F}},s_{\mathrm{Tr}})$ ; shares fixed or evolving (Eq. 3 )	leverage stage

Table 1: Participants in the crash construction. Ranges are sampled per world (Appendix A ). “Active from” refers to the mechanism stages of Figure 3 a; the leverage stage is the first stage used in our laboratories.

Panel of Figure 2 — Figure 2: Episode anatomy. (a) Fixed schedule of a 96-step episode. The utility’s drawdown term is the shaded area between the pre-shock reference and the post-shock reference path (the path drawn is illustrative, not a sampled trajectory). (b) Deterministic order of operations within a step. Shadow-trade settlement (2) and forking (8) run only when the strategy mix evolves; margin checks (5), fundamental and trend traders (7), and the adaptive participant (8) are active from the leverage stage onward (Figure 3 a).

Panel of Figure 3 — Figure 3: Switchable mechanisms. (a) Mechanisms are enabled cumulatively; filled circles mark the active ones. The two laboratories differ only in the outlined last row. (b) The shock’s price decline feeds a liquidation loop (red; strength $\alpha$ ) and a withdrawal loop (teal; strength $\beta$ ). The deleveraging intervention shrinks the leveraged inventory before the shock and so caps the margin deficit; the backstop’s resting bid offsets the thinning of the ladder. (c) With an evolving mix, each step forks both candidate strategies onto exchange copies, marks them one step later, and re-weights the mix ( $\Delta\pi_{i}=\pi_{i}-\min_{k}\pi_{k}$ ).

Condition	Mean regret $\downarrow$	Optimal actions	Scored / planned	Difference from Observe-only
Observe-only	8.878	30/48	48/48	—
Placebo-tools	6.509	33/48	48/48	$-2.369$
Adaptive-lab (primary)	3.785	37/48	48/48	$\mathbf{-5.092}$
Fixed-mix-lab	2.660	44/48	48/48	$-6.218$

Table 2: Prospective results for Luna on 24 newly sampled worlds (48 world–direction units). Optimal-action counts include any feasible utility maximizer. The last column is the paired mean difference from Observe-only ; only Adaptive-lab versus Observe-only is the prespecified primary contrast, and the others are secondary (intervals in Figure 4 b).

Panel of Figure 4 — Figure 4: Laboratory access lowers Luna’s regret relative to observation alone (prospective study, 24 worlds). (a) Mean target regret per condition; parentheses give optimal-action counts. (b) Paired mean differences with nominal 95% stratified world-bootstrap intervals ( Obs = Observe-only , Adapt = Adaptive-lab , Fixed = Fixed-mix-lab , Placebo = Placebo-tools ). The shaded row is the prespecified primary contrast. Among the secondary contrasts, the fixed-mix laboratory also improves on Observe-only , whereas the contrasts against the placebo and between the two laboratories are inconclusive.

Deployment	Observe-only	Adaptive-lab	Fixed-mix-lab	Adapt $-$ Obs [95% int.]	Adapt $-$ Fixed [95% int.]
Sol	6.121 (37)	0.764 (42)	1.091 (46)	$-5.357\;[-8.951,-2.195]$	$-0.327\;[-1.925,\phantom{-}0.786]$
Terra	19.902 (18)	3.550 (40)	4.441 (41)	$-16.352\;[-22.760,-9.686]$	$-0.891\;[-5.934,\phantom{-}4.422]$
Luna	7.676 (36)	4.551 (39)	3.287 (43)	$-3.125\;[-8.181,\phantom{-}2.226]$	$\phantom{-}1.264\;[-3.186,\phantom{-}5.660]$
GPT-5.5 (supp.)	10.017 (31)	1.575 (45)	3.160 (44)	$-8.441\;[-12.393,-4.872]$	$-1.585\;[-4.236,\phantom{-}0.386]$

Table 3: Exploratory mean target regret on 24 previously examined worlds (48 units per cell); parentheses give optimal-action counts out of 48. Intervals are descriptive and were specified after observing outcomes. Terra combines two recorded configurations (Appendix B ). GPT-5.5 is supplementary; no pooled effect is estimated.

Panel of Figure 5 — Figure 5: Exploratory study: laboratory access lowers mean regret for every deployment. Grey bars span Observe-only to Adaptive-lab ; diamonds mark Fixed-mix-lab . Right: post hoc, descriptive world-bootstrap intervals for Adapt $-$ Obs . Worlds were previously examined, so these results motivated, rather than confirm, the prospective study.

Laboratory utility selector	Optimal target actions	Mean target regret
Fixed-mix laboratory $\rightarrow$ adaptive target	47/48	0.3802
Adaptive laboratory $\rightarrow$ adaptive target	46/48	0.5895

Table 4: Exploratory action-ranking diagnostic. Each selector observes the complete laboratory utilities of both interventions and commits before independent target evaluation. Nonzero target regret reflects the remaining transfer errors.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Fixed across worlds
Parameter	Value
Horizon; shock step; intervention step; exposure step	$T=96$ ; $t_{0}=8$ ; step 4; step 1
Tick size; fee; cost scale $\lambda$	1 tick; 1 per lot per counterparty; 100
Fundamental value $v$	10,000 ticks, constant
Risk limits	position $10^{6}$ lots; notional $\leq 20\times$ initial cash
Marketable-order band	100 ticks through the best opposite quote

Appendix

Table 5: Parameters of the crash construction used in both studies. † Sampled and included in the constructor hash (hence in duplicate screening), but the backstop size is fixed at 40 lots in this construction.

Deployment	Condition	Scored	Empty	Blocked	Mean calls (range)
Luna	Observe-only	48	0	0	—
	Adaptive-lab	48	0	0	4.71 (4–7)
	Fixed-mix-lab	48	0	0	4.52 (2–6)
	Placebo-tools	48	0	0	4.25 (3–7)
Sol	Observe-only	25	1	22	—
	Adaptive-lab	22	4	22	4.45 (3–6)

Appendix

Table 6: Prospective census and laboratory use by deployment and condition (48 planned decisions per row). Experiment calls are counted over scored episodes, with a budget of eight.

Value of Information

Value of information studies whether acquiring information can improve decisions enough to justify its cost.

See all 37 papers

MemTrial: Learning When to Trust Memory in LLM Portfolio Agents
Oct 8, 2026Guanghao Wu, Zhuo Cai, Shoujin WangLLM Agent Memory Credit Assignment
University of Technology Sydney
Agentic Trading: When LLM Agents Meet Financial Markets
May 19, 2026Yihan Xia, Panpan You, Taotao Wang +4LLM Agent Evaluation Finance
College of Electronic and Information Engineering, Shenzhen University, Shenzhen, China
Representation Signatures and Risk-Feedback Alignment in LLM Trading Agents
May 16, 2026Weicheng XueLLM Alignment LLM Auditing
Virginia Tech

Oct 8, 2026J/K move · Enter open · S save

MemTrial: Learning When to Trust Memory in LLM Portfolio Agents

Guanghao Wu, Zhuo Cai, Shoujin Wang

University of Technology Sydney

LLM Agent Memory Credit Assignment Portfolio Optimization

Open paper AskPDF arXiv

Large language model (LLM) agents for portfolio management learn from experience: they credit each experience in their memory with the outcome of the decisions that used it. In financial markets, however, this outcome mostly reflects the market move shared by all decisions on that date, so the credit tracks the market rather than the experience, and these agents often do worse than simply holding the equal-weight (1/

N

) portfolio. We ask how an agent can credit an experience with what it changes, and answer it by putting memory on trial: drafts of the same decision with and without an experience face the same market, so the outcome they share cancels in their difference. Our agent, MemTrial, drafts each decision with eight combinations of its retrieved experiences, chosen by a fractional factorial design, and credits each experience with its Banzhaf value, the average of these differences. As each date occurs once and each draft is a noisy LLM sample, these credits are noisy and may not hold on new dates. MemTrial therefore pools them across dates and similar experiences with a hierarchical Bayesian model, acts on them only after they have predicted unseen dates, and otherwise stays anchored at a conservative reference such as 1/

N

. On four benchmarks, MemTrial not only benefits from experiences that matter (the best of 15 methods on a semi-synthetic benchmark with known experience quality) but also limits its losses when its values do not hold (at most 2.2% below 1/

N

on PortBench and InvestorBench, against 15--38% for the best experience-learning agent). Averaged over five settings, it improves the utility of the best experience-learning agent by 21.2%, and with eight LLMs it beats every LLM-based baseline on InvestorBench.

Abstract

Value of Information

Explore similar work

MemTrial: Learning When to Trust Memory in LLM Portfolio Agents

Agentic Trading: When LLM Agents Meet Financial Markets

Representation Signatures and Risk-Feedback Alignment in LLM Trading Agents