Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents
Organizations: Ai For Alpha Neuilly-sur-Seine, France · LAMSADE, Université Paris Dauphine–PSL Paris, France
Abstract
Financial research agents can generate and monitor trading rules faster than a human can document the resulting selection process. Candidates appear at data-dependent times, weak rules are abandoned, and promising ones are watched until they cross a reporting threshold. We introduce PROMOTE, a protocol that assigns every candidate predictable testing wealth at birth and accumulates a cost-aware e-process only from subsequent returns. If lifetime allocations sum to at most one, a weighted Ville argument bounds the probability of ever promoting a rule whose declared bounded net score has always had nonpositive conditional mean. The result permits dependent candidates, adaptive births, and indefinite monitoring. We audit the implementation in two settings. A retrospective pseudo-out-of-sample replay contains 24 panel--rule hypotheses, 3,000 shared-sign null paths, injected alternatives, and public ETF and industry data. Repeated unadjusted tests produce a family crossing in 94.5% of null paths, versus 0.0% for PROMOTE; at a 15-bp aligned ETF-component perturbation, PROMOTE has 70.1% target power and median crossing session 2,327. A separate exact mean-zero experiment lets an agent select candidate identities and birth times from past scores; the funded gate has 1.77% false-family crossings across 3,000 programs. The market replay produces zero promotions after costs. The certificate controls historical false promotion under the declared null; it is not a certificate that an earlier edge remains profitable at deployment. AI-use disclosure. Generative AI tools, including OpenAI Codex, assisted manuscript preparation. For this preprint version, Codex assisted with LaTeX editing, author and venue metadata, compilation, and layout checks.
Figures & tables
| Procedure | Null FWER | 15-bp power | Med. session |
|---|---|---|---|
| Repeated | 94.5 [93.6,95.3] | 99.9 [99.6,100.0] | 125 |
| Fixed | 0.0 [0,.128] | 67.1 [64.7,69.5] | 2,868 |
| Per-stream | 0.7 [.46,1.07] | — | — |
| Promote | 0.0 [0,.128] | 70.1 [67.7,72.3] | 2,327 |
| Shift (ann. %) | Terminal power | Anytime power | |
|---|---|---|---|
| 0 (0.0) | 0.00 [0,.26] | 0.00 [0,.26] | — |
| 5 (4.8) | 0.00 [0,.26] | 0.00 [0,.26] | — |
| 10 (9.6) | 5.73 [4.67,7.03] | 7.73 [6.49,9.20] | 2,568 |
| 15 (14.4) | 67.13 [64.72,69.46] | 70.07 [67.70,72.33] | 2,327 |
| 20 (19.2) | 99.20 [98.61,99.54] | 99.27 [98.69,99.59] | 1,743 |
| 30 (28.8) | 100 [99.74,100] | 100 [99.74,100] | 1,049 |
| Procedure | Crossings | FWER [95% CI] |
|---|---|---|
| Repeated one-sided | 2,792 | 93.07 [92.10, 93.92] |
| Anytime per stream, | 552 | 18.40 [17.05, 19.83] |
| Funded adaptive Promote | 53 | 1.77 [1.35, 2.30] |
| Allocation rule | First | Spent | FWER [95% CI] |
|---|---|---|---|
| Observed-strength split | 25.29 | 93.45 | 1.77 [1.35,2.30] |
| Equal 12-slot reserve | 8.33 | 82.36 | 1.17 [0.84,1.62] |
| Quarter remaining | 25.00 | 93.34 | 1.77 [1.35,2.30] |
| Half remaining | 50.00 | 99.75 | 2.37 [1.88,2.97] |
| Metric | Funded Promote | Naive sleeve | All 24 sleeves |
|---|---|---|---|
| Annualized net return | 0.000% | ||
| Annualized volatility | 0.000% | 0.167% | 1.577% |
| Sharpe, zero cash yield | 0.00 | ||
| Maximum drawdown | 0.00% | ||
| One-way turnover/year | 0.00 | 0.10 | 26.06 |
| Mean gross exposure | 0.00% | 0.55% | 10.23% |
Multiple Hypothesis Testing
Multiple hypothesis testing develops procedures to control false discoveries when many statistical claims are evaluated together.
See all 35 papers