If seven AI advisors agree, what have we learned: that their forecast is reliable, or that the asset is easier to forecast? We examine 928 completed first-quarter outcomes for 232 assets across four historical iPulse AI batches. Ten Gemini configurations form seven equally weighted investment frameworks; no forecasts are regenerated. The council has mean absolute return error of 15.15 percentage points (pp), versus 14.99 for no change, with an unresolved paired difference of +0.16 pp. Error rises from 10.31 to 21.05 pp across disagreement quartiles, but the no-change baseline rises almost identically, from 9.89 to 21.15. In the pooled archive, selecting the lowest-volatility quarter produces lower error than selecting the least-disagreeing quarter: 7.97 versus 10.31 pp. Directional agreement introduces a second trap: its apparent value depends on the aggregation rule. When at least six of seven frameworks share a direction, the sign of their mean return is correct in 53.21% of cases, versus 44.11% with lower agreement. Scoring the modal direction instead changes the lower-agreement accuracy to 54.27%, removing that positive contrast. Recomputed-rank bootstraps, scale-aware losses, and period sensitivities qualify these descriptive results; four overlapping periods cannot establish future-regime performance. Agreement, visible asset risk, and aggregation-specific skill should be evaluated separately before consensus is presented as confidence.
Figures & tables
Framework label
Historical modes retained
Configurations
Paradigm shift
Researcher
1
Contrarian forensic
Researcher
1
Power audit
Researcher
1
Forensic deduction
Researcher
1
Economic machine
Researcher; thinker
2
Civilizational compute
Researcher; thinker
2
Table 1: Fixed historical council, with neutral framework names. Appendix C maps archive labels and explains the perspectives. All configurations use archived Gemini versions; framework diversity is not provider diversity or statistical independence.
Figure 1: Study design. Seven framework-level forecasts are compared with later outcomes and with risk measures available from prices dated before generation. The analysis distinguishes raw error, error relative to a baseline, and selective evaluation.
Figure 2: Overlapping first-quarter outcome windows. Blue circles mark generation dates; gold segments mark the range of target-close observations across assets. Most preceding-quarter labels were unavailable at the next generation. The figure explains why adjusted models are treated as retrospective diagnostics.
Disagreement quartile
Council MAE
No-change MAE
Difference
Mean V
Lowest
10.31
9.89
+0.41
14.31
Second
13.14
13.09
+0.05
16.91
Third
16.10
15.81
+0.29
21.21
Highest
21.05
21.15
−0.10
30.03
Table 2: Within-batch disagreement quartiles, pooled across four batches; 232 outcomes per row. Return errors and horizon-scaled volatility V are in pp. Rounded entries may not subtract exactly.
Figure 3: The raw disagreement–error gradient is also present for no change. A: absolute errors for the council and baseline. B: their paired difference within each quartile. Bars are pointwise asset-cluster 95% intervals.
Figure 4: Visible asset risk and council dispersion. A: 928 outcomes plotted on logarithmic axes. B: within-batch rank correlations. The council’s spread is closely related to historical volatility, which is more strongly related to absolute forecast error than spread is in each batch.
Figure 5: Adjusted and similar-risk comparisons. A: coefficients for standardized transformed predictors in the model of log(1+E) , controlling for batch and asset class. B: lower and higher disagreement within five volatility groups per batch. Bars show pointwise asset-cluster 95% intervals. These are retrospective diagnostics.
Figure 6: Selective evaluation at equal within-batch coverage. A: absolute error among retained forecasts. B: paired error relative to no change. Points are observed estimates at 5% coverage increments; these curves have no uncertainty bands. Paired selector intervals are reported in the text. At 100% all rules retain the same panel. The rules require no fitted outcome labels; their evaluation is retrospective.
Figure 7: Two aggregation rules, two directional conclusions. A: mean-sign and modal-direction accuracy for the same agreement groups; modal ties fall back to mean sign. B: paired council-minus-no-change MAE, rather than visually inferred differences between raw-loss markers. Bars are pointwise asset-cluster 95% intervals, conditional on the observed periods.
Figure 8: Directional association varies across periods and aggregation rules. Each bar is high-minus-lower agreement accuracy within one batch (232 outcomes). These observed-period contrasts have no future-regime uncertainty interpretation.
Figure 9: Sensitivity of the high-minus-low disagreement contrast in excess error. Negative values indicate a smaller council-minus-baseline error in the highest-disagreement quartile. Pointwise asset-cluster 95% intervals span zero across the listed specifications.
Figure 10: Scale-aware diagnostics for the same quartiles. A: mean per-record error divided by historical horizon-scale volatility. B: pooled council skill relative to no change, 1−∑E/∑E0 ; positive favors the council. Points and bars are descriptive estimates without uncertainty bands.
Figure 11: Dependence among framework return forecasts across 928 outcomes. Pearson correlations pool four batches; the companion also supplies matrices within each batch. Neutral names map to historical archive labels in Appendix C . Correlation is not a measure of causal independence or model quality.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Batch
Council MAE
No-change MAE
Difference
Direction
Always up
3
18.88
16.97
+1.91
38.36%
52.16%
4
13.70
14.21
−0.51
50.43%
51.72%
5
13.47
14.08
−0.62
53.88%
59.91%
6
14.54
14.68
−0.14
50.86%
39.22%
Appendix
Table 3: Batch-level estimates in the primary panel, 232 outcomes per row. Return errors are in pp. Direction uses the three-category sign rule; always up is a separate directional baseline.
Disagreement quartile
Council RMSE (pp)
No-change RMSE (pp)
Lowest
17.69
16.55
Second
21.01
20.07
Third
23.39
22.79
Highest
31.14
30.63
Appendix
Table 4: Squared-error sensitivity. Both series exhibit increasing error across disagreement quartiles. This table supplies a second loss function, not a fitted probabilistic forecast evaluation.
Volatility window
Retained outcomes
Council MAE (pp)
Excess MAE (pp)
20 observations
232
7.60
+0.08 [ −0.31 , 0.45]
60 observations
232
7.97
−0.03 [ −0.43 , 0.37]
90 observations
232
7.50
+0.02 [ −0.36 , 0.41]
Appendix
Table 5: Lowest-volatility quarter under alternative historical windows. Brackets are pointwise asset-cluster 95% intervals for the paired error difference relative to no change.
Neutral name
Archive label
Interpretation
Paradigm shift
Elon Musk
Technology adoption, disruption, and execution constraints.
Contrarian forensic
Michael Burry
Downside asymmetry, balance-sheet fragility, and accounting risk.
Power audit
Niccolo Machiavelli
Incentives, control, counterparties, and distribution of economic rents.
Forensic deduction
Sherlock Holmes
Evidence chains, inconsistencies, and competing explanations.
Economic machine
Ray Dalio
Macro cycles, liquidity, debt, and regime transmission.
Civilizational compute
Superintelligence
Compute and energy constraints across long-term system transitions.
Appendix
Table 6: Explanatory framework map. Descriptions summarize configured perspectives, not historical human decisions.
Comparator
MAE (pp)
Excess (pp)
Direction
No change
14.99
0.00
2.48%
Council mean
15.15
0.16
48.38%
Council median
15.13
0.14
51.94%
Repeat prior 60-observation return
25.47
10.49
42.13%
Appendix
Table 7: Unfitted benchmark diagnostics on 928 outcomes. No-change direction is always flat under the three-category sign rule, not the always-up benchmark. Momentum is an intentionally simple comparator with a different lookback length.
Framework
MAE (pp)
Direction
Paradigm shift
15.76
49.78%
Contrarian forensic
17.01
47.84%
Power audit
15.64
51.72%
Economic machine
15.17
50.11%
Forensic deduction
15.93
47.95%
Civilizational compute
15.17
49.78%
Appendix
Table 8: Per-framework point forecasts, averaged across retained modes before scoring. These descriptive comparisons are not model-provider rankings.
Variable (pp unless noted)
Minimum
Median
95th pct.
Maximum
Absolute council forecast
0.01
3.16
11.35
22.24
Realized return
-52.10
0.83
40.57
205.78
Council dispersion
0.22
3.87
9.26
17.83
Horizon-scale historical risk
1.21
17.71
40.45
62.23
Horizon (calendar days)
71.00
88.00
90.00
91.00
Appendix
Table 9: Distribution summaries of the 928 primary records.
Subset
Records
Mean risk
Mean spread
Mean momentum
MAE
Primary balanced
928
20.61
4.41
4.34
15.15
Other valid-risk matched
293
27.78
5.54
7.92
17.00
Appendix
Table 10: Retained versus excluded valid-risk records; variables and MAE are in pp. The excluded set has 293 records across 141 assets, rather than four observations per asset. Differences describe selection and do not establish its cause.
Omitted batch
Mean-sign gap
Modal gap
Excess-error Q4–Q1
3
13.55
2.86
-0.59
4
7.07
-1.89
-0.35
5
6.56
-4.37
-0.59
6
9.41
-0.62
-0.53
Appendix
Table 11: Leave-one-batch-out point sensitivities. Accuracy gaps are high-minus-lower agreement in percentage points; excess-error gaps are pp. These overlapping subsets are not independent validation samples.
Batch
D –risk
Risk–error
D –error
D –excess error
3
0.562
0.482
0.214
-0.026
4
0.641
0.420
0.286
-0.091
5
0.689
0.307
0.236
-0.025
6
0.595
0.371
0.321
0.009
Appendix
Table 12: Within-batch Spearman coefficients in the primary panel. D is numerical disagreement. These rank correlations are descriptive, not causally identified components of uncertainty.
Financial forecasting uses market, company, and economic data to predict prices, returns, volatility, and business outcomes for decisions and risk management.
We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.
Evaluating the true forecasting ability of AI agents requires environments that are resistant to environments resistant to overfitting, free from centralized trust, and grounded in incentive-compatible scoring. Existing benchmarks either rely on static datasets vulnerable to training-data contamination, or measure trading PnL -- a metric conflating predictive accuracy with timing, sizing, and risk appetite. We introduce Foresight Arena, the first permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets. Agents submit probabilistic forecasts on binary Polymarket markets via a commit-reveal protocol enforced by Solidity smart contracts on Polygon PoS; outcomes are resolved trustlessly through the Gnosis Conditional Token Framework. Performance is measured by the Brier Score and a novel Alpha Score -- proper scoring rules that incentivize honest probability reporting and isolate predictive edge over market consensus. We provide a formal analysis: closed-form variance for per-market Alpha, the connection to Murphy's classical Brier decomposition, and a power analysis characterizing the number of rounds required to reliably distinguish agents of different skill levels. We show that detecting a true edge of α∗=0.02 at 80% power requires approximately 350 resolved binary predictions (50 rounds of 7 markets), while α∗=0.01 requires four times more. We complement these analytical results with a deterministic, seed-controlled simulation study calibrated to literature-reported Brier-score ranges, illustrating how Murphy decomposition distinguishes well-calibrated agents from market-tracking agents that fail through reduced resolution. Live results from the deployed benchmark will be reported in a future revision. All smart contracts and evaluation infrastructure are open-source.
Maksym Nechepurenko, Pavel Shuvalov
Director of Research, Devnull, Dubai, UAE · Chief Technology Officer, Devnull, Dubai, UAE
AI financial advisers set a client's stock allocation largely from one stated-risk question, yet people do not answer it the same way twice: in a national US panel, 41% of households chose a different answer two years later, and in a laboratory retest a third of people moved two or more points on a 0-10 scale after six months. Seven large language models from five providers gave one-turn allocation advice for 800 real household profiles in which only the risk answer changed. The recommended stock share moved by 10 to 16 percentage points on average, and by 24 to 35 when the answer changed, against 0.6 to 4.2 points for the same prompt asked twice; in the laboratory retest the rating alone moved advice by 11 to 18 points. Without any note, the models respond to the answer about as strongly as when told that it is very reliable (less so for the two Claude models). If that instructed response is how each model maps attitude to advice, discounting the answer for its measured reliability, as classical test theory prescribes, would cut the swings by a factor of 2.4 to 3.3. The swing is largest for clients under 40. A note that the answer is noisy steadies some models, but shifts their weight to the client's age and pulls cautious clients towards a generic middle allocation: clients unwilling to take any risk are advised 22% stocks instead of 4%. Three extra money questions reduce the swing for some models, with no statistically detectable change relative to how much advice differs across people. In a scripted test, one clarifying question removes nearly all of the swing from an answer the client then corrects (33.5 points to 1.9 on average) while keeping the response to a change the client confirms. We argue that same-person retest audits should be a standard check for AI advisers, and that what to do with an unreliable answer is a question of client autonomy as well as of statistics.