Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors

Bo Qu; Mingguang Chen; Licheng Wang

cs.AISep 22, 2026

Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors

Authors: Bo Qu, Mingguang Chen, Licheng Wang

Organizations: DeepGrounding · AlphaAvatar

Abstract

Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retire them. We ask which of those jobs an agent should keep. Our answer is governed self-evolution: the agent may propose, and a frozen statistical referee that the agent cannot touch must judge. The referee scores each candidate only on market outcomes revealed after submission, by betting, so its false-discovery guarantee holds at every stopping time for any proposal policy. We cross three proposers (a script, a bandit and a language model) with this referee and with three deliberately leaky ones, in a synthetic world with planted truth, a probe-authoring environment and a ten-year walk-forward on the CSI 500. Who judges sets the number of false admissions: the frozen referee admits 5-11 times fewer sub-threshold factors than the leaky referees under a scripted proposer, and no proposer closes that gap. Who proposes sets the yield: the language model beats the script, matches the bandit, and adds the one capability a bandit lacks, writing its own diagnostic probes. The certificate's price is time: an admitted true factor waits about 500 trading days, and the certified portfolio's Sharpe ratio therefore trails an ungated one. Judging belongs to the procedure; proposing and instrument-making belong to the agent.

Figures & tables

Approach	Certifies	Anytime Valid	Adaptive Proposals	Growing Set	Retirement
Backtest Checklist	A batch of backtests	No	No	No	No
Batch Deflation	A batch of trials	No	Partial a	No	No
Agentic Miners	As the two rows above	No	No	No	Heuristic
Certified Self-Modification	The agent’s own modifications	Yes	Partial b	No c	No
This Paper	External hypotheses, scored after submission	Yes	Yes f	Yes d	Yes e

Table 1: What each line of work certifies and when its guarantee holds. “Adaptive proposals” are candidates a learning agent chooses using earlier candidates’ outcomes. “Growing set” means controlling the false-discovery rate over every factor ever admitted. The agentic miners are ( [ ref-alphagen2023 ] ; [ ref-rdagent2025 ] ; [ ref-alphaagent2025 ] ; [ ref-alphaschema2026 ] ). Certified self-modification is ( [ ref-sea2026 ] ; [ ref-pace2026 ] ). Notes: a denotes a fixed batch. b denotes a bounded modification space. c denotes a per-modification budget. d denotes online e-BH. For e , the e-detector guarantees the average run length (§ 4.4 ). f applies to a proposer with no information about post-submission outcomes (§ 4.5 ).

Panel of Figure 1 — Figure 1: The division of labour. The controller (orange) proposes, diagnoses and writes its own instruments. The referee, the data layer and the decision log (blue) are frozen components the controller cannot mount, patch or intercept. The execution layer (grey) turns admitted factors into positions and is never seen by the referee or the controller.

Figure 2: One candidate’s life under the referee, on a simulated stream (an illustration of the mechanics, not a reported result). Top: the 63-day mean of the daily rank-IC of a factor whose edge is

0.03

and then decays to zero, and of a null factor, with the viability threshold

\delta

. Middle: the capital of the admission bet on a log scale. The true factor’s capital reaches the single-candidate bar

1/\alpha

drawn here (the bar as run, over

N_{v}

slots, is higher, § 4.3 ) and the factor is admitted; the null factor’s capital never does. Bottom: the retirement detector, started at admission, stays flat while the edge is intact and climbs to

A

once the edge has decayed.

Panel of Figure 3 — Figure 3: The factorial as a quadrant. Every cell is measured with common random numbers, so the effect of governance (columns) and of intelligence (rows) can be read off separately. The interaction, what the agent adds only when the referee leaks, is a measured quantity.

Referee	False Adm.: Round-Robin	Bandit	LLM (Gemini)	Yield: Round-Robin / Bandit / LLM
Frozen	11.7	9.4	11.7	0.40 / 0.47 / 0.47
Peeking	86.2	38.4	37.4	0.41 / 0.50 / 0.55
Adaptive Threshold	105.2	51.0	43.5	0.39 / 0.44 / 0.48
No Gate	196.0	189.8	78.0	0.37 / 0.37 / 0.59

Table 2: Real data, four-family library: realised false admissions per campaign and realised yield per submission by referee arm and controller. Means cover four start years

\times

five seeds; the LLM column is one model family.

Library	Contrast	Metric	DeepSeek	Gemini	GPT
Four-Family	LLM $-$ Round-Robin	Yield	$++++$ (4/4)	$++++$ (3/4)	$++++$ (2/4)
Four-Family	LLM $-$ Bandit	Yield	$++--$	$+---$	$----$
Nine-Family	LLM $-$ Round-Robin	Yield	$++++$ (4/4)	$++++$ (3/4)	$++++$ (1/4)
Nine-Family	LLM $-$ Bandit	Yield	$+++-$	$+++-$	$----$ (3/4)
Four-Family	LLM $-$ Round-Robin	False Adm.	$+++-$ (0/4)	$-+++$ (0/4)	$++++$ (0/4)
Nine-Family	LLM $-$ Round-Robin	False Adm.	$++++$	$++++$	$++++$

Table 3: Real data, frozen referee: signs of the paired difference between the LLM controller and baseline across the four start years, by model family. Parentheses show the number of starts significant after Holm correction within the family. Families are never pooled.

Family	Frozen: Held / Shelved / bp per Day	No Gate: Held / Shelved / bp per Day
Volatility	473 / 1% / $+4.05$	418 / 2% / $+4.06$
Liquidity	314 / 0% / $+3.79$	555 / 17% / $+3.96$
Short-Horizon Reversal	539 / 30% / $+2.26$	475 / 41% / $+1.50$
Momentum	6 / 83% / $-4.35$	892 / 44% / $+1.69$

Table 4: Per-family sleeve economics under the canonical book, using the round-robin controller and four-family library. Results pool five seeds from the 2016 start. They report sleeves ever held, the share shelved, and each family’s daily earnings while held, gross of their own trading costs . Re-formed daily, a reversal sleeve nets

-19.6

% per year (Figure 4 ).

Figure 4: Persistence and monetisation by family. Left: the decay curve

IC(k)

of a ranking known at

t-1

against the return on day

t+k-1

, with the viability threshold

\delta

. Right: what a single-factor quintile sleeve nets per year at 15 bp per side when re-formed every

h

trading days. Reversal nets at most

+0.9

bp per day at any frequency on the grid; momentum is not certifiable at one day and pays only when held a month or longer.

Figure 5: Cumulative net return of the canonical book, 2016 start, seed-averaged (four-family library; left the round-robin controller, right the LLM controller). The shaded band is the frozen referee’s wait. The ungated book begins compounding in its first year while the certified book is still empty, and is below water until year six. The two slopes are similar once both are invested.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Cost per Side	Certified Net SR	Ungated Net SR
15 bp (registered)	$+0.40$	$+0.73$
25 bp	$+0.25$	$+0.61$
30 bp	$+0.17$	$+0.54$
50 bp	$\mathbf{-0.12}$	$+0.29$

Appendix

Table 5: Cost sensitivity of the two books (five-seed means). The certified book crosses zero between 30 and 50 bp a side. The ungated one does not.

Book	Convention	Net Sharpe	Cumulative Net Return
Certified	Registered	$+0.40\pm 0.03$	$+46.6$ %
Certified	Carry	$+0.40\pm 0.03$	$+47.1$ %
Certified	One-Day Lag	$+0.37\pm 0.03$	$+43.5$ %
Certified	Carry and Lag	$+0.38\pm 0.03$	$+44.1$ %
Ungated	Registered	$+0.73\pm 0.02$	$+66.3$ %
Ungated	Carry	$+0.75\pm 0.02$	$+67.7$ %

Appendix

Table 6: The 2016 round-robin books under four accounting conventions (five-seed means with standard errors, 244-day annualisation). The ungated book’s lead is

0.34

–

0.35

under every convention.

Stream Transformation	Null False Acceptance, %	Median Days to Accept	Detector False Alarms on Alive Streams, %	Step-Decay Delay, Median Days
Raw daily rank-IC (conditional null)	2.1 / 6.0 / 15.4	228 / 227 / 227	1.5 / 11.5 / 46	213 / 181 / 195
Whitened, worst-case bound	1.1 / 1.3 / 1.8	336 / 378 / 462	1.0 / 0.0 / 7.0	250 / 264 / 316
Whitened, predictable bound (adopted)	1.8 / 2.0 / 2.4	238 / 284 / 384	1.5 / 0.5 / 8.5	218 / 234 / 308
Whitened with the true $\rho$ (oracle)	2.1 / 1.3 / 0.9	228 / 344 / 536	1.5 / 0.5 / 6.5	213 / 246 / 333

Appendix

Table 7: Referee ablations use synthetic AR(1) streams at

\rho=0

0.2

0.4

. This table uses cap fraction 0.5. The referee runs at 0.8. The experiment uses ten-year null streams at

\mu=0

and alternative streams at

\mu=0.03

. Alive streams cover 2520 days with run-length target 1260. The nominal level is 5%. Against a Monte-Carlo-calibrated CUSUM, the adopted e-detector’s median delay is 204–322 days by decay shape, against CUSUM’s 181–337.

Family	Referee	Yield: Round-Robin / Bandit / LLM	False Admissions: Round-Robin / Bandit / LLM	Hidden Retries: False
Gemini	Frozen	0.095 / 0.096 / 0.096	0.000 / 0.000 / 0.000	0.000
Gemini	Peeking	0.060 / 0.065 / 0.063	0.314 / 0.300 / 0.315	0.341
Gemini	Adaptive Threshold	0.081 / 0.087 / 0.087	0.272 / 0.273 / 0.263	0.281
Gemini	No Gate	0.042 / 0.034 / 0.043	0.838 / 0.848 / 0.836	0.820
GPT	Frozen	0.095 / 0.096 / 0.096	0.000 / 0.000 / 0.000	0.000
GPT	Peeking	0.060 / 0.065 / 0.061	0.314 / 0.300 / 0.318	0.341

Appendix

Table 8: Synthetic world with planted truth (four families, 1260 days, five seeds). The table reports seed-mean realised yield and false admissions per submission by referee arm and controller for the two LLM families run there. It also reports the hidden-retry attacker’s false admissions.

Family	Fixed Menu: Regret	Authored Probes: Regret	Paired Difference	Holm $p$
DeepSeek	0.616 $\pm$ 0.098	0.228 $\pm$ 0.064	$-0.388\pm 0.101$	0.001
Muse	0.651 $\pm$ 0.084	0.403 $\pm$ 0.064	$-0.248\pm 0.070$	0.002
Gemini	0.692 $\pm$ 0.098	0.458 $\pm$ 0.117	$-0.233\pm 0.085$	0.025
GPT	0.526 $\pm$ 0.078	0.459 $\pm$ 0.097	$-0.067\pm 0.072$	1.000
Hunyuan	0.663 $\pm$ 0.097	0.642 $\pm$ 0.109	$-0.021\pm 0.069$	1.000
GLM	0.585 $\pm$ 0.093	0.659 $\pm$ 0.148	$+0.074\pm 0.114$	1.000

Appendix

Table 9: Probe environment at the registered escalation price. For each model family, the table reports mean intervention regret against the oracle (edge units)

\pm

s.e. over 100 episodes for the fixed diagnosis menu and authored probes. Paired differences are authored minus menu within each episode. Holm correction covers the six evaluable families.

Arm	Admitted	Edge $<0$	$0\leq$ Edge $<\delta$	Edge $\geq\delta$	False Admissions	Retired by the End
Frozen, Round-Robin	214.7	0.0	11.7	203.0	11.7	11.7
Frozen, Bandit	260.8	0.0	9.4	251.3	9.4	9.4
Frozen, LLM (Gemini)	257.0	0.0	11.7	245.3	11.7	11.7
Frozen, LLM (DeepSeek)	258.1	0.0	12.7	245.5	12.7	12.7
Frozen, LLM (GPT)	236.9	0.0	13.3	223.6	13.3	13.3
Peeking, Round-Robin	312.5	10.8	75.3	226.3	86.2	85.6

Appendix

Table 10: Anatomy of admissions on the four-family library (means per campaign over four starts

\times

five seeds). The table splits ever-admitted factors by their realised post-submission mean rank-IC. The first two classes together are realised false admissions. The last column counts false admissions retired by the detector before campaign end.

Referee	Controller	Yield at 0.010 / 0.015 / 0.020	False Admissions at 0.010 / 0.015 / 0.020
Frozen	Round-Robin	0.403 / 0.403 / 0.400	5.4 / 11.7 / 13.8
Frozen	LLM (Gemini)	0.471 / 0.471 / 0.466	5.7 / 11.7 / 14.7
Peeking	Round-Robin	0.407 / 0.406 / 0.402	78.5 / 86.2 / 90.2
Peeking	LLM (Gemini)	0.546 / 0.546 / 0.532	31.9 / 37.4 / 45.7
Adaptive Threshold	Round-Robin	0.390 / 0.390 / 0.387	97.1 / 105.2 / 108.2
Adaptive Threshold	LLM (Gemini)	0.478 / 0.478 / 0.476	37.5 / 43.5 / 47.0

Appendix

Table 11: Label-threshold sensitivity on the four-family library (means over four starts

\times

five seeds). The table reports realised yield per submission and realised false admissions per campaign at label thresholds of

0.010

0.015

(registered), and

0.020

Referee	Controller	Admitted	False: Full	False: After	False at $\delta_{f}$	Edge After	Yield: Full / After
Frozen	Round-Robin	214.7	11.7	17.6	96.0	0.032	0.403 / 0.380
Frozen	Bandit	260.8	9.4	14.4	114.3	0.032	0.474 / 0.460
Frozen	LLM (Gemini)	254.8	11.7	15.8	111.7	0.032	0.466 / 0.458
Peeking	Round-Robin	312.4	86.2	91.2	180.4	0.023	0.406 / 0.390
Peeking	Bandit	342.8	38.4	52.6	170.5	0.027	0.495 / 0.457
Peeking	LLM (Gemini)	354.6	37.4	41.5	152.9	0.029	0.545 / 0.511

Appendix

Table 12: Four-family library results show realised false admissions under three labels. Values are means per campaign over four starts

\times

five seeds. Full is the registered label. After judges each admitted factor only on days after admission.

\delta_{f}

is each family’s break-even (liquidity 0.012, volatility 0.017, momentum 0.031, reversal 0.074).

Referee	Controller	Admitted	False: Full	False: After	Edge After	Yield: Full / After
Frozen	Round-Robin	127.2	9.8	16.6	0.028	0.246 / 0.225
Frozen	Bandit	188.1	11.9	17.6	0.030	0.357 / 0.340
Frozen	LLM (Gemini)	198.6	14.6	20.3	0.030	0.377 / 0.364
Peeking	Round-Robin	256.9	127.3	131.8	0.016	0.244 / 0.233
Peeking	Bandit	295.8	97.0	107.2	0.020	0.360 / 0.330
Adaptive Threshold	Round-Robin	296.4	175.2	174.9	0.013	0.230 / 0.228

Appendix

Table 13: Nine-family library results show the same measures. They exclude a per-family break-even because it is undefined for the five fundamental families.

Controller	Referee	Admitted	False: Full	False: After	Share False	Edge After	Yield	Wait
Round-Robin	Frozen (Recorded)	214.7	11.7	17.6	0.08	0.032	0.403	439
Round-Robin	Fixed 500 (Daily)	208.0	16.4	22.2	0.11	0.029	0.358	500
Round-Robin	Fixed 500 (At End)	207.2	16.0	21.8	0.11	0.029	–	–
Round-Robin	Fixed 250 (Daily)	187.4	16.6	19.4	0.10	0.030	0.314	250
Bandit	Frozen (Recorded)	260.8	9.4	14.4	0.06	0.032	0.474	408
Bandit	Fixed 500 (Daily)	262.6	12.2	20.4	0.08	0.028	0.441	500

Appendix

Table 14: Four-family library results show fixed-horizon comparator referees run as counterfactual programs on the frozen cell’s recorded submission stream. Values are means per campaign over four starts

\times

five seeds. Each tests the post-submission mean once, 500 or 250 days after submission. Each applies Benjamini–Hochberg at 0.05 whenever a window closes ( Daily , deployable) or once at the campaign’s end ( At End , so no yield or wait).

Controller	Referee	Admitted	False: Full	False: After	Share False	Edge After	Yield	Wait
Round-Robin	Frozen (Recorded)	127.2	9.8	16.6	0.12	0.028	0.246	543
Round-Robin	Fixed 500 (Daily)	150.2	41.4	49.7	0.33	0.021	0.214	500
Round-Robin	Fixed 500 (At End)	148.8	40.8	48.9	0.33	0.021	–	–
Round-Robin	Fixed 250 (Daily)	115.0	30.2	33.8	0.29	0.023	0.159	250
Bandit	Frozen (Recorded)	188.1	11.9	17.6	0.09	0.030	0.357	436
Bandit	Fixed 500 (Daily)	207.8	33.2	44.0	0.22	0.024	0.330	500

Appendix

Table 15: Nine-family library results show the same comparators.

Start	No Gate	Frozen	Frozen, Retirement Off	Retirement Off Minus Frozen	Fixed 500 (Daily)	Fixed 500 Minus Frozen
2016	$+0.74\pm 0.02$	$+0.40\pm 0.03$	$+0.41\pm 0.03$	$+0.00\pm 0.00$	$+0.60\pm 0.01$	$+0.20\pm 0.03$
2017	$+0.78\pm 0.01$	$+0.46\pm 0.01$	$+0.47\pm 0.01$	$+0.01\pm 0.00$	$+0.59\pm 0.02$	$+0.13\pm 0.02$
2018	$+0.75\pm 0.01$	$+0.42\pm 0.02$	$+0.42\pm 0.02$	$+0.01\pm 0.00$	$+0.65\pm 0.02$	$+0.24\pm 0.02$
2019	$+0.62\pm 0.04$	$+0.49\pm 0.05$	$+0.50\pm 0.05$	$+0.01\pm 0.00$	$+0.61\pm 0.02$	$+0.12\pm 0.04$

Appendix

Table 16: Net Sharpe of counterfactual books on the round-robin frozen cell’s candidate sequence. Results use the four-family library and five seeds per start. They report means with standard errors and differences paired per seed. Retirement off holds every factor admitted by the frozen referee through the campaign’s end. Fixed 500 (Daily) is Table 14 ‘s deployable 500-day comparator. It is held from its admission day until the runs’ e-detector retires it.

Family	Contrast	Yield Diff. Pre-2020	Yield Diff. Post-2020	False Adm. Diff. Pre-2020	False Adm. Diff. Post-2020
Gemini	LLM $-$ Round-Robin	$+0.009\pm 0.005$	$+0.088\pm 0.004$	$-0.2\pm 0.1$	$+0.1\pm 0.2$
Gemini	LLM $-$ Bandit	$+0.026\pm 0.006$	$-0.016\pm 0.004$	$+0.3\pm 0.2$	$+1.9\pm 0.7$
DeepSeek	LLM $-$ Round-Robin	$+0.028\pm 0.009$	$+0.097\pm 0.004$	$-0.1\pm 0.1$	$+1.1\pm 0.6$
DeepSeek	LLM $-$ Bandit	$+0.045\pm 0.011$	$-0.007\pm 0.004$	$+0.5\pm 0.2$	$+2.8\pm 0.9$
GPT	LLM $-$ Round-Robin	$+0.008\pm 0.003$	$+0.050\pm 0.004$	$-0.1\pm 0.1$	$+1.7\pm 0.4$
GPT	LLM $-$ Bandit	$+0.025\pm 0.006$	$-0.054\pm 0.003$	$+0.5\pm 0.2$	$+3.4\pm 0.8$

Appendix

Table 17: Controller contrasts on the four-family library under the frozen referee, split by candidates submitted before and from 2020-01-02. Each campaign has about 114 and 308 such candidates. LLM-minus-baseline differences are paired by start and seed. Values are means

\pm

s.e. over 20 pairs.

Family	Retired	Share Retired	Days Held	IC While Held	IC After	After $<\delta$	After $<0$
Liquidity	12.1	0.24	207	$-0.0053$	$+0.0040$	1.00	0.05
Momentum	0.5	1.00	354	$-0.0131$	$+0.0035$	0.89	0.22
Reversal	31.8	0.36	838	$+0.0145$	$+0.0338$	0.00	0.00
All	44.3	0.21	661	$+0.0088$	$+0.0254$	0.28	0.01

Appendix

Table 18: Retirement outcomes on real data using the four-family library, frozen referee, and round-robin controller. Results cover four starts

\times

five seeds. They report factors retired per campaign and family. For retired factors, results also report mean daily rank-IC during the held window and the remaining campaign. These factors had at least 21 days held and 21 days left in the campaign.

Start	Frozen	No Gate	(a)	(b)	Breadth	Selection	Earliness
2016	$+0.40$	$+0.74$	$+0.63$	$+0.72$	$+0.02$	$+0.09$	$+0.23$
2017	$+0.46$	$+0.78$	$+0.65$	$+0.75$	$+0.04$	$+0.10$	$+0.19$
2018	$+0.42$	$+0.75$	$+0.67$	$+0.71$	$+0.04$	$+0.04$	$+0.25$
2019	$+0.49$	$+0.62$	$+0.53$	$+0.57$	$+0.05$	$+0.04$	$+0.04$

Appendix

Table 19: Net Sharpe of counterfactual books under the canonical construction. The books are frozen, ungated, (a) the referee’s factors on the ungated schedule, and (b) a fixed random subset of ungated factors matching the referee’s count. Breadth is no gate less (b). Selection is (b) less (a). Earliness is (a) less frozen. The three sum to the gap. Sharpe here uses 252 days.

Figure 6: Realised false admissions per campaign by referee arm and controller (real data, four-family library; means over four start years

\times

five seeds; log scale). The referee sets the count; the controller moves it by a factor of 2–2.5, and only when the referee leaks.

Figure 7: Admission delay of realised-true factors by referee arm (LLM controller, four-family library, 2016 start, five seeds pooled). The no-gate spike is the pre-registered 126-day incubation.

Family	Certified: Sleeves	First Held (Mean Day)	After 2020	Ungated: Sleeves	First Held (Mean Day)	After 2020
Growth	42	1151	100%	235	1365	83%
Investment	0			412	1419	76%
Liquidity	126	1511	100%	213	1264	83%
Momentum	0			354	1561	84%
Quality	45	1177	100%	306	1541	86%
Reversal	237	1722	100%	164	1266	100%

Appendix

Table 20: Nine-family library with the round-robin controller, 2016 start, and five pooled seeds. The table reports sleeves ever held by the book. It also reports each family’s mean first-held campaign day, of 2,589. Finally, it reports the share of holding windows with midpoints after 2020-01-02, campaign day 975.

Family	Start	LLM $-$ Round-Robin Yield	Holm $p$	LLM $-$ Bandit Yield	Holm $p$
DeepSeek	2016	$+0.095\pm 0.007$	0.006	$+0.039\pm 0.007$	0.124
DeepSeek	2017	$+0.079\pm 0.005$	0.005	$+0.007\pm 0.004$	1.000
DeepSeek	2018	$+0.080\pm 0.010$	0.034	$-0.004\pm 0.006$	1.000
DeepSeek	2019	$+0.070\pm 0.006$	0.012	$-0.002\pm 0.007$	1.000
Gemini	2016	$+0.064\pm 0.009$	0.122	$+0.008\pm 0.007$	1.000
Gemini	2017	$+0.070\pm 0.005$	0.008	$-0.002\pm 0.002$	1.000

Appendix

Table 21: Paired realised-yield contrasts under the frozen referee on the four-family library. Contrasts compare the LLM controller with round-robin and the bandit. They report the mean difference across five common-random-number seeds and its standard error. They also report the Holm-adjusted two-sided

p

within the model family, over 24 tests.

Family	Start	LLM $-$ Round-Robin Yield	Holm $p$	LLM $-$ Bandit Yield	Holm $p$
DeepSeek	2016	$+0.142\pm 0.004$	0.000	$+0.029\pm 0.007$	0.178
DeepSeek	2017	$+0.120\pm 0.009$	0.003	$+0.011\pm 0.008$	1.000
DeepSeek	2018	$+0.148\pm 0.007$	0.001	$+0.026\pm 0.004$	0.072
DeepSeek	2019	$+0.090\pm 0.009$	0.009	$-0.010\pm 0.005$	1.000
Gemini	2016	$+0.163\pm 0.003$	0.000	$+0.050\pm 0.005$	0.012
Gemini	2017	$+0.134\pm 0.007$	0.001	$+0.026\pm 0.006$	0.260

Appendix

Table 22: The same contrasts on the nine-family library.

Controllers	Model Families	Referee Arms	Libraries	Starts $\times$ Seeds	Cells
Round-Robin, Bandit	None	All Four	Both	$4\times 5$	320
LLM	Gemini, DeepSeek, GPT	Frozen	Both	$4\times 5$	120
LLM	Gemini	Peeking, Adaptive Threshold, No Gate	Four-Family	$4\times 5$	60
LLM	DeepSeek, GPT	No Gate	Four-Family	$4\times 5$	40
Total					540

Appendix

Table 23: Coverage of the real-data factorial. Every row covers four start years and five seeds. The last row is the confirmatory extension run after the factorial.

Id	Date	What Changed	Affects
A1	2026-09-02	LLM costs metered at billed prices; spend cap $335	No (accounting)
A2	2026-09-02	Deterministic iteration order; longer retry window	No (engineering)
A3	2026-09-03	Escalation-price sensitivity computed by re-scoring recorded episodes	No (analysis)
A4	2026-09-04	Abstract effort pins realised through the measured vendor map	No (runtime)
A6	2026-09-04	Probe environment audited; two world defects fixed (world v2)	Probe Environment
A7	2026-09-05	Real-data world specification on the CSI 500	Real-Data World

Appendix

Table 24: Dated amendments to the pre-registered design. A5 and A9 were never issued. Every book amendment used zero-cost re-booking of the recorded cells (Proposition 2).

Rung	Allocation Across Families	Response to an Alarm	Instruments	Inputs Beyond Family Statistics
Round-robin (C0)	Cycles through families	Scripted: retire and refill the same family	Fixed menu	None
Discounted-UCB bandit (C1)	Reward per family: admissions less retirements, discounted	Scripted, as C0	Fixed menu	None
LLM controller (C2)	Reads statistics, family profiles and memory	Diagnosis (idiosyncratic death, crowding, regime shift, data breakage) or escalation; chooses the response	Authors, tests and mounts probes	Family profiles, memory, the reasons for retirements

Appendix

Table 25: The controller ladder. Every rung uses the same library, submission budget and referee. The rungs differ in policy and, at the top rung, inputs and actions. Thus, the contrasts compare controller packages.

Family	Signal (sign)	Parameters Drawn	Specs
Momentum	$\log(P_{t-1-s}/P_{t-1-s-L})$ ( $+$ )	$L\in[40,250]$ , skip $s\in[0,21]$	4,642
Short-Horizon Reversal	$-\log(P_{t-1}/P_{t-1-L})$ ( $+$ after sign)	$L\in[3,30]$ ; turnover-scaled or not	56
Volatility	$-$ (std, downside std or max) of daily returns over $L$ ( $+$ after sign)	$L\in[10,130]$ ; three kinds	363
Liquidity	$-$ mean turnover over $L$ , or Amihud illiquidity ( $+$ after sign)	$L\in[10,130]$ ; two kinds	242
Value	Book, earnings, sales or dividends to price, smoothed over $L$ ( $+$ )	$L\in[1,21]$ ; four kinds	84
Size	$-\log$ market capitalisation, circulating or total ( $+$ after sign)	$L\in[1,21]$ ; two kinds	42

Appendix

Table 26: The nine factor families (the four-family library is the first four). Prices use adjusted closes. Fundamentals are announcement-aligned. The grid is enumerable, fixing the hypothesis universe before the campaign.

Quantitative Finance

Quantitative finance uses mathematical models, statistics, and machine learning to value assets, measure risk, and guide trading and portfolio decisions.

See all 191 papers

Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research
Oct 21, 2026Houde Qian, Zheng Jiang, Wenyi Li +9Financial Language Model Evaluation Algorithmic Trading
Navers Lab, Einsia.AI
InvestPhilBench: A Multi-Layer Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy
Jun 24, 2026Mingguang Chen, Bo QuLLM Evaluation Finance
University of California, Riverside (UCR). · 2Illinois Institute of Technology (IIT).
AQuA: Recursively Self-Improving Quantitative Trading Research Agents
Aug 13, 2026Jiacheng Guo, Suozhi Huang, Yunlong Gao +5Quantitative Finance Automated Algorithm Discovery
Princeton University · Ant Group · Stanford University

Oct 21, 2026J/K move · Enter open · S save

Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research

Houde Qian, Zheng Jiang, Wenyi Li, Chenxi Xu+8

Navers Lab, Einsia.AI

Financial Language Model Evaluation Algorithmic Trading Quantitative Finance AI Agent Benchmarks

Open paper AskPDF arXiv

Quantitative research is an iterative loop: a researcher proposes a market hypothesis, implements it as an executable signal, tests it on historical data, and decides what to keep. Existing benchmarks score formulas, code, or individual trading decisions, leaving open whether large language models (LLMs) can complete this loop and produce strategies that work beyond the research period. We introduce Quant-Arena, a benchmark and public leaderboard for end-to-end quantitative research on real A-share market data. A model constructs features and stock-scoring code, receives feedback only on 2022--2023, and freezes one strategy for assessment on the sealed year 2024. A cash-account evaluator applies trading constraints and costs, including market impact estimated from a simulated order book. Scores are dominated by realized excess return and complemented by signal quality and the diversity of features used by the strategy. We evaluate nine frontier models, among which GPT-6 Astra scores highest; yet no model beats the equal-weight market in 2024. Quant-Arena is open to submissions from any model or agent harness.

Abstract

Quantitative Finance

Explore similar work

Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research

InvestPhilBench: A Multi-Layer Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy

AQuA: Recursively Self-Improving Quantitative Trading Research Agents