The Only Edge Was Memorized in Training: LLM Trading Agents Around Earnings and Bad News
Abstract
Large language model (LLM) trading agents are tested on news, but news reaches stock prices within minutes, and a benchmark that replays headlines at their publication time finds no alpha in trading on them. Money can then remain in two places, before news whose date is known, such as earnings announcements, and after bad news, if the market's fall went too far. However, studies of LLMs and earnings read documents published with or after the announcement, and studies of reversal sort monthly portfolios, so neither tells whether an agent can trade these moments after costs. In this paper, we test LLM agents at both moments on the same look-ahead-free benchmark, before 598 earnings announcements and after 409 bad-news drops of large US stocks. Every input is cut at the decision close, staged agents are compared with single calls, simple rules and a logistic regression, and every decision is made with and without the stock's name. Across five models from four vendors, no agent beat the simple bets. Before earnings, the agents mostly bought and earned what buying before every announcement earned, and after bad news, a Gemini-based agent called almost every fall an overreaction while no agent picked better dips than buying them all. The only gains came from what the models had memorized in training. Given only the ticker and the dates, a GPT-based model called the direction of 59.2% of the announcements before its training cutoff and 53.3% after it. Agent builders should therefore compare an event agent with the simple bet and test it blind after the training cutoff.
Figures & tables
| Control | Decision |
|---|---|
| Buy every drop | Buys all events (a pure reversal bet). |
| Never buy | Holds nothing. |
| Event-type rule | Buys only after an earnings or analyst-rating story; fixed before any agent result. |
| Logistic regression | Buys when the predicted chance of beating the 56-stock average over 20 days is above one half. Inputs: event type, drop, trends, sector and market move, headline counts, and whether a bad legal, executive or guidance headline appeared in the previous ten days. Trained on two periods and tested on the third. |
| One-call agent | Reads the headlines once and decides. |
| Random same-sector stocks | Replaces each bought stock with a random other stock of its sector on the same day (the null of rta-bench-1). |
| Period | Events | Before open | After close | In session | Drops |
|---|---|---|---|---|---|
| 2020 | 243 | 134 | 83 | 26 | 9 |
| 2022 | 242 | 126 | 86 | 30 | 29 |
| 2026 | 205 | 114 | 55 | 36 | 26 |
| All | 690 | 374 | 224 | 92 | 64 |
| Sample | 2020 | 2022 | 2026 | All |
|---|---|---|---|---|
| Bad news, drop 3% (main) | 139 | 155 | 115 | 409 |
| Bad news, drop 2% | 210 | 254 | 182 | 646 |
| Bad earnings, any drop | 53 | 51 | 22 | 126 |
| No news, drop 3% (sample) | 244 | 217 | 139 | 600 |
| Before earnings | After bad news | |||
| Model | Runs | Calls | Runs | Calls |
| GPT-5.6 Luna | V0, V2–V4, probe; main sample | 7,774 | both agents; no-news control | 6,190 |
| Gemini 3.5 Flash | V0–V3 | 8,280 | both agents | 4,990 |
| Claude Haiku 4.5 | V0, V2, V3 | 6,900 | both agents | 4,995 |
| Llama 3.2 3B | V0, V2, V3 | 6,900 | both agents | 4,399 |
| Claude Sonnet 4.6 | V0, V2, V3; 49 clean events | 490 | four calls, named, 2026 | 527 |
| What we found | Common expectation | What to do | |
|---|---|---|---|
| V-A | Buying before every announcement paid off only in 2020 | Buying before earnings earns a steady premium | Report the always-long rule next to the agent |
| V-A | Before earnings, the agents mostly just bought | An agent that reasons before earnings picks a side | Check what share of the agent’s trades are purchases |
| V-A | No agent beat the five controls, and a logistic regression did at least as well | Reasoning before earnings beats simple models | Test against always-long and a logistic regression |
| V-B | The agents called the direction of the reaction about as well as a coin | An LLM can tell a good report from a bad one in advance | Report accuracy against the bar for significance |
| V-B | Steps, peers, short selling and insider filings did not improve the trades | More reasoning and more inputs help the agent | Add a step or an input only when a paired test shows a gain |
| V-B | The drop warning was no better than volatility unless the agent knew the stock | An LLM spots large drops ahead of time | Compare a drop warning with a ranking by volatility |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Run | Trades | Long | Acc. (%) | AUC good | AUC drop | Net (bp) | Alpha (bp) | ||
| C1 always long | 598 | 598 | – | – | – | +18.1 | +0.86 | +12.2 | +0.57 | |
| C2 momentum (20 days) | 598 | 315 | – | – | – | +6.3 | +0.33 | +1.2 | +0.07 | |
| C3 peer return | 523 | 281 | – | – | – | +5.8 | +0.32 | +16.6 | +0.92 | |
| C4 logistic regression | 598 | 348 | 53.8 | 0.53 | 0.54 | +22.1 | +1.11 | +26.1 | +1.31 | |
| GPT-5.6 Luna | named V0 | 91 | 91 | 53.3 | 0.55 | 0.58 | +4.2 | +3.10 | +3.9 | +3.14 |
| named V2 | 270 | 263 | 53.7 | 0.54 | 0.59 | +5.4 | +1.64 | +4.3 | +1.31 | |
| Model | Run | V3 C1 | V3 C4 | V3 V0 | V3 V2 | V4 V3 |
|---|---|---|---|---|---|---|
| GPT-5.6 Luna | named | 8.3 [ 47.9, +30.7] | 22.1 [ 61.8, +17.0] | 0.0 [ 5.2, +5.2] | 0.4 [ 4.7, +3.8] | +0.2 [ 2.9, +3.4] |
| blind | 11.0 [ 50.8, +27.8] | 24.9 [ 65.8, +15.6] | +0.2 [ 6.4, +7.1] | +0.7 [ 3.7, +5.1] | 1.7 [ 5.5, +2.0] | |
| Gemini 3.5 Flash | named | +2.9 [ 34.3, +39.6] | 10.9 [ 54.4, +32.0] | 2.0 [ 19.9, +16.0] | +2.7 [ 3.5, +9.7] | – |
| blind | 12.6 [ 50.7, +24.9] | 26.5 [ 71.9, +17.9] | 8.8 [ 30.1, +12.3] | +2.4 [ 5.1, +10.3] | – | |
| Claude Haiku 4.5 | named | 1.8 [ 38.1, +33.6] | 15.7 [ 56.0, +24.2] | +3.0 [ 8.6, +14.4] | 4.0 [ 10.5, +2.1] | – |
| blind | 3.3 [ 40.7, +33.5] | 17.1 [ 58.3, +24.3] | +3.3 [ 7.1, +13.7] | 0.8 [ 6.2, +4.5] | – |
| Model | Run | Before cutoff | After cutoff | Clean test |
|---|---|---|---|---|
| GPT-5.6 Luna | V0 | +2.7 [ 1.2, +6.8] | +3.7 [ 4.9, +13.8] | +1.0 [ 6.5, +9.6] |
| V2 | +3.2 [ 4.4, +10.6] | +5.9 [ 13.3, +24.4] | +11.6 [ 2.2, +24.8] | |
| V3 | +1.8 [ 4.1, +7.5] | +6.1 [ 8.1, +22.7] | +1.4 [ 10.4, +14.2] | |
| Gemini 3.5 Flash | V0 | +7.2 [+0.3, +14.2] | +12.4 [+3.4, +24.4] | +1.6 [ 5.7, +9.1] |
| V2 | +26.0 [+11.3, +41.4] | 12.2 [ 38.6, +13.7] | 23.5 [ 41.3, 5.2] | |
| V3 | +30.3 [+16.5, +46.2] | 22.0 [ 44.9, +0.2] | 41.1 [ 64.7, 17.2] |
| Before earnings (bp) | After bad news (pp) | |||
|---|---|---|---|---|
| Model | Before cutoff | After cutoff | Before cutoff | After cutoff |
| GPT-5.6 Luna | +1.8 [ 4.1, +7.5] | +6.1 [ 8.1, +22.7] | +0.18 [ 0.57, +1.04] | +1.22 [ 1.58, +3.97] |
| Gemini 3.5 Flash | +30.3 [+16.5, +46.2] | 22.0 [ 44.9, +0.2] | +0.10 [ 0.29, +0.63] | +0.11 [ 0.21, +0.58] |
| Claude Haiku 4.5 | +3.1 [ 7.5, +14.1] | 2.7 [ 28.2, +21.5] | +0.66 [ 0.71, +2.09] | +2.43 [+0.22, +5.52] |
| Llama 3.2 3B | +5.8 [ 0.3, +12.9] | +5.7 [ 2.2, +14.3] | +1.70 [ 4.08, +8.53] | 4.17 [ 8.70, 0.19] |
| Method | Run | Bought | Mean (%) | Diff (pp) | Rank | Alpha (%) | AUC further fall | |||
| Buy every drop | 409 | +0.94 | +2.02 | – | – | 98 | +7 | +0.85 | – | |
| Event-type rule | 144 | +1.83 | +2.10 | +1.40 | +1.12 | 98 | +53 | +2.49 | – | |
| Logistic regression | 213 | +1.22 | +1.58 | +0.59 | +0.52 | 97 | +5 | +0.40 | 0.45 [0.39, 0.49] | |
| GPT-5.6 Luna | 4 calls, named | 296 | +1.30 | +2.33 | +1.32 | +0.97 | 99 | +15 | +1.70 | 0.60 [0.54, 0.66] |
| 4 calls, blind | 285 | +0.92 | +1.85 | 0.07 | 0.05 | 90 | +9 | +0.83 | 0.55 [0.49, 0.62] | |
| 1 call, named | 84 | 0.56 | 0.59 | 1.90 | 1.66 | 22 | 3 | 0.18 | 0.62 [0.57, 0.67] |
| Method | Run | 2020 | 2022 | 2026 | Clean test |
|---|---|---|---|---|---|
| Buy every drop | +1.85 (139) | +0.33 (155) | +0.61 (115) | 0.36 (53) | |
| Event-type rule | +1.25 (53) | +1.71 (65) | +3.31 (26) | +1.43 (5) | |
| Logistic regression | +1.10 (85) | +1.19 (74) | +1.48 (54) | +2.06 (25) | |
| GPT-5.6 Luna | 4 calls, named | +2.32 (108) | 0.16 (116) | +2.10 (72) | +1.69 (36) |
| 4 calls, blind | +1.91 (100) | 0.05 (113) | +0.99 (72) | +0.14 (35) | |
| 1 call, named | +0.58 (46) | 1.30 (22) | 2.97 (16) | 5.33 (8) |
| Method | Run | Bought | Mean | Alpha | AUC | |
|---|---|---|---|---|---|---|
| Buy every drop | 126 | +0.67 | +0.95 | +1.48 | – | |
| Logistic regression | 21 | 1.46 | 0.74 | 1.02 | 0.44 | |
| GPT-5.6 Luna | 4 calls, named | 69 | +1.08 | +1.27 | +2.20 | 0.65 |
| 4 calls, blind | 67 | +0.78 | +0.87 | +1.22 | 0.60 | |
| 1 call, named | 34 | +1.74 | +1.68 | +1.96 | 0.64 | |
| 1 call, blind | 13 | +0.73 | +0.31 | +0.27 | 0.58 |
Algorithmic Trading
Algorithmic trading uses computational models to generate signals, allocate portfolios, and execute orders across financial markets.
See all 167 papers