LLM Trading Agents Earn Beta, Not Alpha: A Look-Ahead-Free Benchmark
Abstract
Large language model (LLM) trading agents now trade stocks for their users, and benchmarks rank models by their agents' returns. Yet a single run's return says little about skill. In our study, two runs of the same model that differed only in the random seed ended 5.2 points apart on average, so telling skill from luck takes many backtests across markets, models and seeds. However, live arenas trade each day only once, and backtests either use the few months after a model's training cutoff or run on years the model may remember, often with news joined to prices by calendar date. In this paper, we present rta-bench-1, a backtest over three US markets (a rally, a bear market and a crash). It shows each headline only after its publication minute, plays each period blind and named, and checks every run for look-ahead. It also splits each return into factor exposure and alpha and scores every decision against random choices of the same day and sector. In 91 runs of six models from four vendors, no alpha was significant, market and style exposure accounted for a median 78% of the return, and the decisions did no better than random stocks of the same sector. The agents bought the stocks with the most positive news, yet in ablations they earned as much without their headlines. Still, the agents carried out rules reliably, since a single cash rule in the prompt cut a GPT-based agent's share in stocks by 30.3 points in all six pairs. Builders should therefore supply the edge themselves and report exposures and alpha, and benchmarks should replay news by publication time.
Figures & tables
| SPY (%) | Equal | |||
|---|---|---|---|---|
| Period | Days | Return | Drawdown | weight (%) |
| 2026 rally | 217 | +12.1 | 8.9 | +11.4 |
| 2022 bear market | 251 | 18.6 | 24.5 | 12.4 |
| 2020 crash, rebound | 253 | +17.2 | 33.7 | +40.1 |
| Model | Vendor | Cutoff | 2026 |
|---|---|---|---|
| GPT-5.6 Luna | OpenAI | Feb 2026 | partly |
| Gemini 3.5 Flash | Jan 2025 | clean | |
| Claude Haiku 4.5 | Anthropic | Jul 2025 | clean |
| Claude Sonnet 4.6 | Anthropic | Jan 2026 | partly |
| Claude Haiku 5.5 | Anthropic | Jun 2026 | partly |
| Llama 3.2 3B | Meta | Dec 2023 | clean |
| 2026 rally | 2022 bear market | 2020 crash and rebound | ||||||||||
| Runs | Return | Beta | Alpha | Runs | Return | Beta | Alpha | Runs | Return | Beta | Alpha | |
| GPT-5.6 Luna | 6 | +7.8 | 0.35 | +3.5 | 6 | 3.7 | 0.29 | 3.0 | 6 | +0.9 | 0.30 | 8.4 |
| Gemini 3.5 Flash | 4 | +6.3 | 0.70 | 1.6 | 4 | 8.4 | 0.46 | 8.9 | 4 | +13.7 | 0.47 | 9.2 |
| Haiku 4.5 | 1 | +7.2 | 0.59 | 1.2 | 1 | 5.3 | 0.58 | 1.0 | 1 | +16.9 | 0.50 | 2.3 |
| Sonnet 4.6 | 1 | +8.1 | 0.59 | +1.8 | 1 | +0.4 | 0.55 | 0.5 | 1 | +25.4 | 0.34 | +3.6 |
| Haiku 5.5 | 1 | +2.4 | 0.23 | 3.2 | 1 | +1.4 | 0.01 | 0.0 | 1 | +7.4 | 0.23 | 2.6 |
| Ablation | Pairs | Return | Decision |
|---|---|---|---|
| No news | 18 | 0.3 [ 3.4, +2.7] | +0.09 [ 0.08, +0.24] |
| Swapped news | 18 | 2.2 [ 5.3, +1.1] | +0.15 [ 0.05, +0.38] |
| Calendar date | 18 | 1.4 [ 4.4, +1.9] | +0.05 [ 0.08, +0.19] |
| All decisions | After-close news | |||
|---|---|---|---|---|
| News | Open | Close | Open | Close |
| Publication time | 0.04 | 0.00 | 0.15 | +0.06 |
| ( 0.4) | (0.0) | ( 0.4) | (+0.2) | |
| Calendar date | +0.02 | +0.08 | 0.00 | +0.53 |
| (+0.2) | (+0.6) | (0.0) | (+1.5) | |
| Measure | No news | Swapped news | Calendar date |
|---|---|---|---|
| Share in stocks | +3.8 | 1.8 | +0.7 |
| [ 2.2, +9.6] | [ 5.2, +1.6] | [ 2.7, +4.0] | |
| Worst five-day loss | +0.8 | +0.6 | +1.1 |
| [+0.2, +1.5] | [ 0.4, +1.4] | [+0.4, +1.8] | |
| Maximum drawdown | +1.1 | +1.8 | +2.0 |
| [ 0.4, +2.8] | [ 0.2, +3.7] | [+0.4, +3.4] |
| 2026 | 2022 | 2020 | ||||
|---|---|---|---|---|---|---|
| News | Cal. | Pub. | Cal. | Pub. | Cal. | Pub. |
| Earnings, bad | +1.26 | +0.22 | +0.93 | +0.02 | +0.17 | +0.03 |
| Earnings, good | 0.01 | 0.03 | +0.25 | +0.20 | +0.63 | 0.01 |
| Guidance, bad | +1.11 | +0.06 | +0.33 | 0.03 | +0.33 | +0.04 |
| Guidance, good | +0.10 | 0.27 | +0.54 | +0.38 | +0.94 | +0.34 |
| Model | Earnings | Other good | No good | Ratio |
|---|---|---|---|---|
| GPT-5.6 Luna | 7.6 | 2.8 | 2.3 | 2.7 |
| Gemini 3.5 Flash | 12.5 | 4.9 | 1.2 | 2.6 |
| Haiku 4.5 | 22.6 | 10.5 | 2.9 | 2.2 |
| Sonnet 4.6 | 12.3 | 4.4 | 2.1 | 2.8 |
| Haiku 5.5 | 1.7 | 1.1 | 0.2 | 1.5 |
| News | Events | Whole | First hour | Rest | Next open |
|---|---|---|---|---|---|
| Earnings and guidance | |||||
| Outside regular hours | 90 | +1.64 | +0.68 | +0.85 | +0.71 |
| (2.6) | (2.4) | (1.7) | (1.7) | ||
| Regular hours | 53 | 0.31 | 0.08 | 0.33 | 0.37 |
| ( 0.6) | ( 0.7) | ( 0.7) | ( 1.3) | ||
| All | 143 | +0.92 | +0.40 | +0.41 | +0.31 |
| 2026 | 2022 | 2020 | ||||
|---|---|---|---|---|---|---|
| Leak | Fixed | Dec. | Fixed | Dec. | Fixed | Dec. |
| None (as shipped) | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| Calendar-date news | 0/4 | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 |
| Next day’s return | 0/4 | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 |
| Times 4 hours early | 0/4 | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 |
| Dates 3 days early | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 |
| Prompt | Return | Share in stocks | Drawdown | Volatility | Selection |
|---|---|---|---|---|---|
| All four risk rules | 7.5 | 28.1 | 8.0 | 10.1 | 3.4 |
| [ 23.1, +6.2] | [ 34.9, 21.2] | [ 14.9, 1.5] | [ 16.0, 5.9] | ||
| 25% cap per stock | 8.1 | 7.6 | 3.0 | 4.5 | 5.2 |
| [ 20.0, +0.0] | [ 14.6, 1.1] | [ 7.8, +1.1] | [ 8.9, 0.5] | ||
| 30% cash when risk-off | 5.6 | 30.3 | 9.4 | 10.8 | 1.1 |
| [ 19.4, +4.5] | [ 36.2, 24.6] | [ 14.5, 5.7] | [ 16.5, 6.9] |
| What we found | Common expectation | What to do | |
|---|---|---|---|
| V-A | The agents sold near the bottom of the 2020 crash and bought back late | – | Set cash and exposure limits in the prompt and the account |
| V-A | A trend-following rule made more than every agent | An LLM agent beats a simple rule | Compare each agent with SPY, an equal-weight portfolio and a trend rule |
| V-A | No run had a significant alpha | An agent that beats the market has skill | Report alpha and exposure with every return |
| V-A | Two seeds of one model ended more than half as far apart as the best and the worst model | One run per model is enough to rank models | Run several seeds before any ranking |
| V-B | The trades were no better than random stocks of the same sector, and the agents sold too early | A profitable agent makes good trades | Score each trade against random choices |
| V-B | The agents bought the stocks with the best news, and the news score predicted nothing | The news an agent favors predicts returns | Report what the agent trades on |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Period | Set | Seed | Return | Beta | Alpha | Alpha | Decision | Decisions |
|---|---|---|---|---|---|---|---|---|---|
| Haiku 4.5 | 2020 | leaderboard | 0 | +16.9 | 0.50 | 2.3 | 0.2 | +0.09 | 204 |
| Haiku 4.5 | 2022 | leaderboard | 0 | 5.3 | 0.58 | 1.0 | 0.2 | 0.01 | 240 |
| Haiku 4.5 | 2026 | leaderboard | 0 | +7.2 | 0.59 | 1.2 | 0.1 | 0.70 | 218 |
| Haiku 5.5 | 2020 | leaderboard | 0 | +7.4 | 0.23 | 2.6 | 0.6 | 1.01 | 44 |
| Haiku 5.5 | 2022 | leaderboard | 0 | +1.4 | 0.01 | 0.0 | 0.0 | +0.66 | 4 |
| Haiku 5.5 | 2026 | leaderboard | 0 | +2.4 | 0.23 | 3.2 | 0.5 | 1.13 | 66 |
| Model | Period | Set | Seed | Return | Beta | Alpha | Alpha | Decision | Decisions |
|---|---|---|---|---|---|---|---|---|---|
| Haiku 4.5 | 2020 | leaderboard | 0 | +12.8 | 0.55 | 3.7 | 0.4 | +0.11 | 253 |
| Haiku 4.5 | 2022 | leaderboard | 0 | 9.5 | 0.48 | 7.6 | 1.0 | 0.31 | 278 |
| Haiku 4.5 | 2026 | leaderboard | 0 | +8.8 | 0.57 | 1.5 | 0.1 | 0.60 | 261 |
| Sonnet 4.6 | 2026 | leaderboard | 0 | +8.5 | 0.70 | +1.5 | +0.2 | 0.13 | 165 |
| Gemini 3.5 Flash | 2020 | leaderboard | 0 | +18.6 | 0.52 | 11.2 | 0.9 | +0.36 | 73 |
| Gemini 3.5 Flash | 2020 | leaderboard | 1 | +9.3 | 0.53 | 15.9 | 1.4 | 0.13 | 93 |
| News | Events | Whole | First hour | Rest | Next open |
|---|---|---|---|---|---|
| Earnings and guidance | |||||
| Outside regular hours | 225 | +1.48 | +0.83 | +0.55 | +0.41 |
| (4.1) | (3.8) | (1.97) | (1.6) | ||
| Regular hours | 116 | 0.27 | 0.02 | 0.36 | 0.33 |
| ( 0.9) | ( 0.2) | ( 1.4) | ( 1.7) | ||
| All | 341 | +0.88 | +0.54 | +0.24 | +0.16 |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,304 papers