Inside the Trading Loop: Markets, Models, and Moves with LLM Agents
Organizations: R2M AI, Cornell University · R2M AI, Carnegie Mellon University · Shanghai Jiao Tong University · University of California San Diego
Abstract
We built a hybrid multi-agent trading system that combines large language model (LLM) analysts with momentum-based portfolio construction and rule-based execution. Company-news and macroeconomic agents return structured reports to a coordinator that sets stock weights and cash reserves. We evaluate the portfolio engine using its broker-recorded paper account, which operated with deterministic fallbacks, and compare LLM-enabled configurations through retrospective replays. From March 11 to August 28, 2026, the account returned 34.5%, compared with 14.4% for the S&P 500 fund SPY over the same dates. This raw return advantage came with a larger maximum drawdown. Reconstructed policies also finish above matched SPY and QQQ references when assessments are used in the same session or deferred to the next. Design comparisons examine cash authority, selective invocation, and response contracts. A constant-cash control gives a return estimate close to stance-based sizing, and shorter report contracts improve observed parsing but can change recommendations.
Figures & tables
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Observation | Evidence | Disposition | Recorded status |
|---|---|---|---|
| Provenance is required | Model and fallback must be distinguishable per call | Log answer source, parse status, fallback reason | Implemented |
| Prompt invites unsupported context | Replies mention indicators absent from the supplied inputs | Remove the unsupported-context instruction | Implemented |
| Contract makes coverage selective | Failure reaches 27.7% on the richest logged inputs; stance changes with the contract | Shorten sentiment replies; revalidate the macro stance | Validated, not yet deployed |
| Placement blocks current membership | Max name-weight change 0.09 pp; current selection ignores sentiment | Disable sentiment as a scoring input; test candidate discovery | Scheduled, not yet deployed |
| Margin localizes cash effects | 98% of allocation-change magnitude at on the later 59 dates | Validate margin-gated invocation on observable calls | Pending prospective validation |
| Sizing changes exposure | Similar point estimate under constant cash | Withhold stance sizing; require exposure and action-set controls | Withheld |
| Program group | Role and dependency boundary |
|---|---|
| parse_logs.py , reliability.py | Extract model/fallback provenance and summarize available attempt/task records. |
| run_agents.py , ablation_schema.py | Generate historical prompt outputs and contract variants; rerunning requires API access and provider charges. |
| reconstruct_l0.py , replay.py | Reconstruct baseline and intervention trajectories; require restricted inputs and the private strategy. |
| audit_public_results.py | Recompute retained target-allocation and routing diagnostics without private code or network access. |
| consumer_topology.py | Run the self-contained synthetic placement comparison; no market data or model calls. |
| Claim | Evidence level |
|---|---|
| Parse failure rises with prompt length | Logged attempts; shortened-contract test on a stratified sample |
| Macro stance is concentrated on one action, under the production contract | Directly measured over 233 parsed samples |
| Allocation transmission | Historical reconstruction; tested under a 3 sentiment-score error model |
| Boundary concentration | In-window diagnostic; threshold intervention; generic pipeline |
| Return differences | Counterfactual estimates with intervals |
| # | Decision point | Class |
|---|---|---|
| 1 | Universe membership | c a d |
| 2 | Technical sub-scores | a |
| 3 | Trend / regime / RS / VIX | a |
| 4 | Alpha factors | a |
| 5 | Headline sentiment | c (b fallback) |
| 6 | Composite score / grade | a |
| Day class | Cash changed | Book dist. (pp) | |
|---|---|---|---|
| Event (calendar or earnings) | 47 | 29.8% | 2.21 |
| Non-event | 72 | 23.6% | 1.67 |
| Salience-gated ( ) | 66 | 25.8% | 1.90 |
| Not gated | 53 | 26.4% | 1.86 |
| Margin-closest 55% | 66 | 45.5% | 3.24 |
| Remaining 45% | 53 | 1.9% | 0.19 |
| Event | Count | Rate |
|---|---|---|
| Reconstruction, sentiment component (30,846 tasks, 2 samples) | ||
| Parse failure (logged API attempts, ) | 334 | 6.9% |
| Stopped at max_tokens =500 (logged subset) | 379 | 7.9% |
| Final parse failure (all prompt-sample tasks) | 131 | 0.4% |
| Reconstruction, macro component (238 tasks, 2 samples) | ||
| Final parse failure / API error (all tasks) | 1 / 4 | 0.4 / 1.7% |
| Cell (119 run dates each) | Parsed | Trunc. | scale_in / hold | |
|---|---|---|---|---|
| Haiku 4.5, production (replay, sample 1) | 117 | n/a | 113 / 4 | — |
| Haiku 4.5, instruction removed | 98 (+21) | 2 | 116 / 3 | 0.96 |
| Sonnet 5, production (thinking on) | 0 (+10) | 118 | 10 / 0 | — |
| Sonnet 5, thinking off, same cap | 91 (+28) | 0 | 107 / 12 | 0.87 |
| Haiku 4.5, forced distribution over stances | 88 (+30) | 4 | 91 / 27 | 0.96 |
| Haiku 4.5, minimal contract | 119 | 0 | 59 / 60 | 0.96 |
| Fallback | Sign | sent | comp | Half- | Max | Flips |
|---|---|---|---|---|---|---|
| Lexicon (deployed) | 52% | 16.0 | 3.8 | 0.098 | 0.086 | 0/119 |
| Neutral (no signal) | 99% | 17.3 | 4.5 | 0.092 | 0.087 | 0/119 |
| FinBERT | 29% | 14.4 | 3.3 | 0.070 | 0.082 | 0/119 |
| Cash | Invested | ret. vs. L0 (pp) | 95% CI |
|---|---|---|---|
| 0% | 100% | ||
| 5% | 95% | ||
| 10% | 90% | ||
| 15% | 85% | ||
| 20% | 80% |
| Materiality threshold | 0 | 0.5 pp | 1 pp | 2 pp |
|---|---|---|---|---|
| L1 | 119 | 31 | 31 | 31 |
| L2 | 119 | 79 | 79 | 79 |
| Constant 5% cash | 85 | 85 | 85 | 85 |
| Noise ( MAE) | Flips | Cash | vs L0 | vs L1 | Return vs L0 (pp) |
|---|---|---|---|---|---|
| 0 (L1) | 0 | 31 | 1.88 | 0 | |
| 1 | 0 | 31 | 1.89 | 0.04 | , |
| 2 | 0 | 31 | 1.90 | 0.08 | , |
| 3 | 0 | 31 | 1.91 | 0.11 | , |
| Policy | Signal | Calls | Cash dates | Allocation change retained |
|---|---|---|---|---|
| All macro dates | — | 100% | 31/31 | 100% |
| Input-based gate | input | 55% | 17/31 | 56% |
| Random, equal budget | none | 55% | — | 40–71% |
| Margin, closest 55% | consumer | 55% | 30/31 | 95% |
| Margin | consumer | 66% | 31/31 | 98% |
| Dates | Calls/d | $ | Cash d. | Ret. | (pp) | 95% CI | |
|---|---|---|---|---|---|---|---|
| 1 † | 29 | 32 | 12 | 15 | 48% | — | — |
| 2 † | 43 | 47 | 18 | 21 | 66% | — | — |
| 3 (L1) | 60 | 66 | 25 | 27 | 86% | ||
| 3 (macro only) | 60 | 0.50 | 0.8 | 27 | 85% | ||
| 4 (L1) | 79 | 87 | 33 | 31 | 98% | ||
| 4 (macro only) | 79 | 0.66 | 1.1 | 31 | 96% |
| Calibration | Evaluation | Calls | Transmission | Cash dates | |
|---|---|---|---|---|---|
| 40 | 79 | 4 | 62.0% | 96.3% | 13/13 |
| 50 | 69 | 4 | 63.8% | 96.7% | 12/12 |
| 60 | 59 | 4 | 67.8% | 97.8% | 12/12 |
| 70 | 49 | 4 | 77.6% | 98.6% | 11/11 |
| (points) | Date-shift pairs | Cash differs | Share |
|---|---|---|---|
| 98 | 51 | 52% | |
| 109 | 29 | 27% | |
| 90 | 21 | 23% | |
| 229 | 17 | 7% | |
| 280 | 6 | 2% | |
| 13 | 0 | 0% |
| Arm (Table 18 ) | Return (%) | Sharpe | Max DD (%) |
|---|---|---|---|
| L0 Quant-only | 59.9 | 2.09 | |
| L1 Advisory | 59.0 | 2.06 | |
| L1 sentiment-only | 59.7 | 2.09 | |
| L1 macro-only | 59.2 | 2.06 | |
| FinBERT at the L1 interface | 59.6 | 2.08 | |
| L2 Sizing | 63.0 | 2.11 |
| Comparison | What is varied | Main readouts |
|---|---|---|
| Model substitution | Model versus deterministic fallback with the same market record and rules | Selected stocks, cash, target-allocation distance |
| Rule interventions | Score coefficients, cash boundaries, invocation gates; synthetic placement | Dependence on rule settings, changes retained per request budget |
| Cash policies | Model-directed sizing versus constant-cash controls | Exposure, paired returns, uncertainty |
| Response formats | Contract and token cap on retained historical inputs | Parsing, usable fields, score and action stability |
| Agent role | Allocation effect | Cost | |||
|---|---|---|---|---|---|
| Arm | permitted intervention | Days pp allocation change | Mean allocation distance (pp) | Prompts/date | $ |
| L0 Quant-only (operational rule) | nothing | 0 | 0 | 0 | 0 |
| L1 Advisory (as designed) | sentiment coefficient 0.15; half of macro | 31 | 1.9 | 131 | 50 |
| L1 sentiment-only | sentiment only | 0 | 0.1 | 130 | 48 |
| L1 macro-only | macro only | 31 | 1.8 | 1 | 2 |
| L2 Sizing | L1 + stance cash 0–40% | 79 | 4.4 | 131 | 50 |
| One run date ahead | Five run dates ahead | |||
|---|---|---|---|---|
| Signal | Ticker-days | Accuracy [95% CI] | Ticker-days | Accuracy [95% CI] |
| LLM sentiment | 14,371 | 49.4 [47.4, 51.4] | 12,764 | 50.1 [47.1, 53.2] |
| Lexicon | 10,672 | 49.4 [48.1, 50.8] | 9,679 | 49.3 [47.0, 52.0] |
| FinBERT | 14,460 | 49.2 [48.1, 50.2] | 12,848 | 49.5 [47.8, 51.2] |
| Arm vs. L0 | return (pp) | 95% CI |
|---|---|---|
| L1 Advisory | ||
| L2 Sizing | ||
| Constant 5% cash |
| Consumer | Changes | Swaps | Dist. | ||
|---|---|---|---|---|---|
| A threshold | 1 | 0.15 | 2.1% | — | — |
| B top- , pre-selection | 1 | 0.15 | 54.9% | 0.62 | 6.4 |
| C top- , post-selection | 0 | 0.15 | 0 | 0 | 0.57 |
| Series | Return | Max drawdown | Sharpe |
|---|---|---|---|
| Recorded account window: March 11–August 28 | |||
| Paper account | +34.5 | -26.8 | 1.46 |
| SPY | +14.4 | -6.3 | 2.10 |
| QQQ | +18.2 | -11.2 | 1.66 |
| Initial basket | +18.5 | -11.2 | 1.67 |
| Same-session reconstruction: March 12–August 31 | |||
| Policy | Same-session reconstruction | Next-session assessments |
|---|---|---|
| L0 | +56.38 | +72.01 |
| L1 | +55.54 | +71.05 |
| L2 | +58.72 | +78.30 |
| Constant 0% | +61.32 | +81.99 |
| Constant 5% | +58.95 | +77.14 |
| Constant 10% | +56.44 | +72.89 |
| Policy | Difference | Paired-date interval | Five-date block interval |
|---|---|---|---|
| L1 Advisory | -0.84 | ||
| L2 Sizing | +2.34 | ||
| Cash 0% | +4.94 | ||
| Cash 5% | +2.57 | ||
| Cash 10% | +0.06 | ||
| Cash 15% | -3.69 |