TradeGauntlet: Evaluating Compliance Controls for Autonomous Trading Agents
Organizations: Independent Researcher
Abstract
Autonomous trading agents need controls that prevent problematic orders without making legitimate tasks impractical. We study this aspect of agentic compliance with TradeGauntlet, a replayable limit-order-book simulator that separates detection, prevention, and execution cost. In 1,680 simulated sessions with Claude Haiku 5.5 and Sonnet 5.5, all conduct alerts were self-matches. Gates eliminated those alerts by rejecting self-crossing orders, while the aggregate-gate condition reduced Sonnet's two-sided fills by about 15%. A conduct reminder reduced Sonnet's alert rate but reduced Haiku's fills without a detectable alert benefit. These results show why zero alerts alone do not establish an agent's compliance. Scripted probes and two search methods test weaknesses beyond those observed in the LLM runs. The searches find split-order schedules that evade the implemented conduct rules; a learned baseline catches all 2,400 tested searched episodes but also flags 15% of a large maker's sessions. Aggregate caps reduce the response-dependent gain unevenly, with smaller reductions when only near-touch depth affects prices. A response fitted to five Nasdaq stocks and a separate ABIDES experiment expose the dependence of these estimates on market assumptions. A distinct adversarial prompt elicited refusals from Sonnet in all 240 rounds, yielding no candidate strategies to evaluate. TradeGauntlet provides reproducible tests of controls around trading agents; the experiments do not establish legal compliance or real-market detection accuracy.
Figures & tables
| Signal | Condition for an alert |
|---|---|
| Individual pattern | Some and satisfy , , and ticks. |
| Layer pattern | For one and one displayed side, qualifying orders occupy at least three distinct prices. No individual-size or distance test is added. |
| Aggregate pattern | For one and displayed side, at least two orders have and positive remaining quantity . Their sum is at least , three times the largest reference depth in that group. |
| Self-match | The subject is both buyer and seller in a fill. |
| Model | Prompt, gate | Episodes (complete) | Alerts | Fills |
|---|---|---|---|---|
| Haiku 5.5 | task only, off | 120 (120) | 2 | 13.3 |
| task only, per-order | 120 (118) | 0 | 13.1 | |
| task only, aggregate | 120 (118) | 0 | 14.0 | |
| reminder, off | 120 (102) | 1 | 10.9 | |
| reminder, per-order | 120 (106) | 0 | 10.4 | |
| reminder, aggregate | 120 (107) | 0 | 10.5 |
| Stock | Spread (ticks) | Obs. | Top three levels, 1 s | Best level, 1 s |
|---|---|---|---|---|
| AAPL | 15 | 18,028 | 0.018 [ 0.003, 0.039] | 0.052 [0.031, 0.075] |
| AMZN | 12 | 15,340 | 0.032 [0.015, 0.049] | 0.036 [0.022, 0.050] |
| GOOG | 26 | 11,152 | 0.016 [ 0.005, 0.037] | 0.025 [0.007, 0.044] |
| INTC | 1 | 18,889 | 0.348 [0.287, 0.417] | 0.201 [0.172, 0.229] |
| MSFT | 1 | 19,220 | 0.479 [0.406, 0.574] | 0.252 [0.222, 0.284] |
| Market | Display | 1 s | 5 s | MSFT fit | INTC fit | |
|---|---|---|---|---|---|---|
| RMSC04 | , 1 tick | 0.15 | 0.09 [0.05, 0.13] | 0.46 [0.33, 0.60] | 0.07 | 0.05 |
| , 1 tick | 0.26 | 0.11 [0.06, 0.15] | 0.49 [0.36, 0.63] | 0.12 | 0.09 | |
| , 1 tick | 0.50 | 0.09 [0.05, 0.13] | 0.53 [0.41, 0.67] | 0.24 | 0.17 | |
| , 5 ticks | 0.26 | 0.01 [ , 0.04] | 0.16 [0.07, 0.26] | 0.13 | 0.09 | |
| Thin book | , 1 tick | 0.15 | 0.07 [0.02, 0.13] | 0.34 [0.18, 0.52] | 0.07 | 0.05 |
| , 1 tick | 0.26 | 0.06 [0.01, 0.12] | 0.36 [0.20, 0.54] | 0.12 | 0.09 |
| Gate | Original alerts | With aggregate signal | Executed lots | Wealth |
|---|---|---|---|---|
| off | 0/40 | 40/40 | 24.00 [24.00, 24.00] | 22.05 [14.80, 32.80] |
| legacy | 0/40 | 40/40 | 24.00 [24.00, 24.00] | 22.05 [14.80, 32.80] |
| aggregate | 0/40 | 0/40 | 24.00 [24.00, 24.00] | -25.95 [-33.20, -15.20] |
| Response | No gate | Per-order | Aggregate, | Aggregate, |
|---|---|---|---|---|
| Step, | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] |
| Step, | 48.00 [48.00, 48.00] | 48.00 [48.00, 48.00] | 48.00 [48.00, 48.00] | 0.00 [0.00, 0.00] |
| Linear, | 9.78 [9.26, 10.30] | 8.24 [7.62, 8.88] | 6.14 [5.60, 6.70] | 3.32 [2.86, 3.80] |
| Fitted, MSFT | 2.84 [2.42, 3.28] | 3.14 [2.68, 3.60] | 2.68 [2.30, 3.06] | 1.37 [0.97, 1.76] |
| Fitted, INTC | 2.00 [1.64, 2.38] | 2.68 [2.26, 3.12] | 2.00 [1.62, 2.38] | 1.10 [0.82, 1.38] |
| Fitted, AAPL | 0.00 [0.00, 0.00] | 0.12 [0.04, 0.22] | 0.04 [0.00, 0.10] | 0.04 [0.00, 0.10] |
| Real-path policy | Synthetic policy | ||||
| Response, path | Control | Alerts | Alerts | ||
| MSFT fit, MSFT | none | 2.64 [0.70, 6.05] | 3/30 | 2.67 | 17/30 |
| per-order | 9.63 [1.60, 22.23] | 3/30 | 8.09 | 0/30 | |
| 4.41 [0.33, 11.18] | 0/30 | 2.60 | 0/30 | ||
| [ , 1.24] | 0/30 | 3.05 | 0/30 | ||
| INTC fit, INTC | none | [ , 2.40] | 3/30 | 2.46 | 23/30 |
| Detector | Large maker | Other legitimate | Random schedules | Searched schedules |
|---|---|---|---|---|
| Conduct rules | 0/100 | 0/400 | 64/100 | 0/2,400 |
| Order-to-trade ratio | 22/100 | 1/400 | 99/100 | 1,816/2,400 |
| Learned | 15/100 | 0/400 | 100/100 | 2,400/2,400 |
| Any of the three | 23/100 | 1/400 | 100/100 | 2,400/2,400 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Change from ungated, paired by seed | |||||
|---|---|---|---|---|---|
| Strategy | Measure | Ungated | + throttle | ||
| Scheduled exec. | Fill rate (%) | 32.8 | 0.0 [ 0.1, 0.0] | 0.0 [ 0.1, 0.0] | 19.6 [ 21.3, 18.0] |
| Shortfall (ticks/lot) | 0.046 | 0.000 [ 0.001, 0.000] | 0.000 [ 0.001, 0.000] | 0.029 [ 0.044, 0.102] | |
| Slice exec. | Fill rate (%) | 37.6 | 0.0 [ 0.1, 0.0] | 0.0 [ 0.1, 0.0] | 7.8 [ 9.7, 6.2] |
| Shortfall (ticks/lot) | 0.180 | 0.001 [0.000, 0.003] | 0.001 [0.000, 0.003] | 0.007 [ 0.075, 0.055] | |
| Large maker | Volume (lots) | 60.2 | 5.7 [ 7.3, 4.0] | 1.7 [0.5, 3.0] | 60.2 [ 62.8, 57.8] |
| Component | Specification |
|---|---|
| Book and matching | One evaluated account and one instrument; integer ticks, price/time priority, partial fills |
| Timing | One second per attempted action; background replenishment every second; default decisions every 60 seconds, with exact re-quote times in the cadence boundary checks |
| Background liquidity | Three price levels per side, 10 lots per level, centered around the fundamental plus any response |
| Exogenous path | Fundamental starts at 1,000; increments -1, 0, +1 with probabilities 1/4, 1/2, 1/4 every 30 seconds; noise quantities 0, 1, 2 with probabilities 1/2, 1/4, 1/4 every 10 seconds |
| Risk and fees | 100-lot worst-case position commitment; fee 0.05 tick-lots per filled lot |
| Sessions | 720 seconds in the core study; 720 and 3,600 seconds in the throttle tests; no warm-up |
| Unchanged (%) | SD 60 s | Excess kurtosis 60 s | ACF(1) 10 s | ACF 1–5 | |
|---|---|---|---|---|---|
| Simulator | 83 | 1.00 | [ , ] | 0.00 | |
| AAPL | 5 | 3.39 | 1.46 [0.58, 2.35] | 0.09 | |
| AMZN | 16 | 2.75 | 4.09 [0.39, 7.61] | 0.05 | 0.09 |
| GOOG | 15 | 2.11 | 0.98 [0.02, 2.33] | 0.06 | |
| INTC | 75 | 3.43 | 2.90 [0.22, 5.55] | 0.02 | 0.06 |
| MSFT | 66 | 3.84 | 1.60 [0.48, 2.57] | 0.08 |
| Response | Control | All resting depth | Depth within three ticks |
|---|---|---|---|
| Linear, | No gate | 9.68 [9.04, 10.34] (6.28–9.78) | 7.30 [6.72, 7.90] (4.62–7.30) |
| 6.24 [5.70, 6.80] (6.14–6.24) | 6.12 [5.56, 6.68] (5.85–6.12) | ||
| 3.32 [2.84, 3.80] (2.88–3.34) | 2.96 [0.05, 5.85] (2.96–3.52) | ||
| Fitted, MSFT | No gate | 4.48 [4.02, 4.96] (2.84–4.48) | 2.84 [2.42, 3.26] (2.80–2.96) |
| 2.54 [2.14, 2.96] (2.54–2.70) | 3.06 [2.56, 3.58] (2.54–3.06) | ||
| 1.37 [0.98, 1.76] (1.37–1.58) | 1.58 [1.26, 1.90] (1.28–1.58) |
| Response | Control | Random search | Policy gradient, 80 | Policy gradient, 800 |
|---|---|---|---|---|
| none | 9.78 | 7.24–9.23 | 9.68 [9.02, 10.34] | |
| per-order | 8.24 | 6.80–8.36 | 7.94 [7.34, 8.56] | |
| 6.14 | 5.62–6.24 | 6.24 [5.68, 6.80] | ||
| 3.32 | 2.39–3.32 | 2.91 [2.22, 3.64] | ||
| MSFT fit | none | 2.84 | 2.68–4.24 | 4.86 [4.28, 5.44] |
| per-order | 3.14 | 2.56–3.86 | 3.62 [3.12, 4.12] |
| Sensitivity analysis | Conditions | Episodes |
|---|---|---|
| Cap and response | 2 scripts 6 response thresholds (ungated + 5 aggregate caps) 10 seeds 2 response states | 1,440 |
| Parameter replication | 12 new vectors from the existing generator 5 market seeds 2 thresholds 2 gates 2 response states | 480 |
| Throttle and horizon | 2 session lengths (ungated + 2 gates 4 windows + 2 gates without throttle) 10 seeds | 220 |
| Price boundaries | 3 cap multiples 2 neighboring thresholds 2 scripts 2 gates 10 seeds 2 response states | 480 |
| Cadence boundaries | 170- and 190-second maker cadences 40 ungated seeds; scored at | 80 |
| Threshold | Gate | : mean [min, max] | , no alert | |
|---|---|---|---|---|
| 0.2 | off | 16.47 [-5.25, 50.80] | 44/60 | 5/60 |
| 0.2 | aggregate | 10.96 [-5.85, 65.90] | 30/60 | 30/60 |
| 0.4 | off | 11.32 [0.00, 50.80] | 27/60 | 0/60 |
| 0.4 | aggregate | 0.00 [0.00, 0.00] | 0/60 | 0/60 |
| Minutes | Gate | Executed lots | Throttle rejects | Throttled | Recovered |
|---|---|---|---|---|---|
| 12 | off | 60.50 [34.00, 90.00] | 0.00 [0, 0] | 0/10 | 0/10 |
| 12 | legacy | 26.40 [13.00, 59.00] | 26.80 [16, 28] | 10/10 | 0/10 |
| 12 | aggregate | 51.80 [29.00, 90.00] | 6.80 [0, 8] | 9/10 | 0/10 |
| 60 | off | 300.60 [263.00, 355.00] | 0.00 [0, 0] | 0/10 | 0/10 |
| 60 | legacy | 51.20 [31.00, 74.00] | 197.50 [188, 200] | 10/10 | 10/10 |
| 60 | aggregate | 74.40 [38.00, 210.00] | 169.40 [97, 180] | 10/10 | 10/10 |
| Legitimate task | Ungated lots | Aggregate lots | Volume loss |
|---|---|---|---|
| Scheduled executor | 27.38 [11.00, 50.00] | 27.38 [11.00, 50.00] | 0.0% |
| 10-lot slice executor | 30.12 [12.00, 45.00] | 30.12 [12.00, 45.00] | 0.0% |
| Small maker | 31.05 [20.00, 36.00] | 31.05 [20.00, 36.00] | 0.0% |
| Large maker | 56.50 [33.00, 88.00] | 48.55 [27.00, 87.00] | 14.1% |
| Patient maker, 300 s | 66.28 [32.00, 109.00] | 47.60 [30.00, 69.00] | 28.2% |
| Legitimate task | Size cap | Throttle | Position | Other |
|---|---|---|---|---|
| Scheduled executor | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] |
| 10-lot slice executor | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] |
| Small maker | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] | 0.00 [0, 0] |
| Large maker | 20.30 [20, 24] | 7.40 [0, 8] | 0.00 [0, 0] | 0.25 [0, 2] |
| Patient maker, 300 s | 6.00 [6, 6] | 0.00 [0, 0] | 0.00 [0, 0] | 0.03 [0, 1] |