When AI Advisors Agree: Distinguishing Forecast Risk from Forecast Reliability

Russlan Ramdowar

cs.MAOct 9, 2026

When AI Advisors Agree: Distinguishing Forecast Risk from Forecast Reliability

Authors: Russlan Ramdowar

Organizations: Future Edge Group FZE, United Arab Emirates · iPulse AI

Abstract

If seven AI advisors agree, what have we learned: that their forecast is reliable, or that the asset is easier to forecast? We examine 928 completed first-quarter outcomes for 232 assets across four historical iPulse AI batches. Ten Gemini configurations form seven equally weighted investment frameworks; no forecasts are regenerated. The council has mean absolute return error of 15.15 percentage points (pp), versus 14.99 for no change, with an unresolved paired difference of +0.16 pp. Error rises from 10.31 to 21.05 pp across disagreement quartiles, but the no-change baseline rises almost identically, from 9.89 to 21.15. In the pooled archive, selecting the lowest-volatility quarter produces lower error than selecting the least-disagreeing quarter: 7.97 versus 10.31 pp. Directional agreement introduces a second trap: its apparent value depends on the aggregation rule. When at least six of seven frameworks share a direction, the sign of their mean return is correct in 53.21% of cases, versus 44.11% with lower agreement. Scoring the modal direction instead changes the lower-agreement accuracy to 54.27%, removing that positive contrast. Recomputed-rank bootstraps, scale-aware losses, and period sensitivities qualify these descriptive results; four overlapping periods cannot establish future-regime performance. Agreement, visible asset risk, and aggregation-specific skill should be evaluated separately before consensus is presented as confidence.

Figures & tables

Framework label	Historical modes retained	Configurations
Paradigm shift	Researcher	1
Contrarian forensic	Researcher	1
Power audit	Researcher	1
Forensic deduction	Researcher	1
Economic machine	Researcher; thinker	2
Civilizational compute	Researcher; thinker	2

Table 1: Fixed historical council, with neutral framework names. Appendix C maps archive labels and explains the perspectives. All configurations use archived Gemini versions; framework diversity is not provider diversity or statistical independence.

Figure 1: Study design. Seven framework-level forecasts are compared with later outcomes and with risk measures available from prices dated before generation. The analysis distinguishes raw error, error relative to a baseline, and selective evaluation.

Figure 2: Overlapping first-quarter outcome windows. Blue circles mark generation dates; gold segments mark the range of target-close observations across assets. Most preceding-quarter labels were unavailable at the next generation. The figure explains why adjusted models are treated as retrospective diagnostics.

Disagreement quartile	Council MAE	No-change MAE	Difference	Mean $V$
Lowest	10.31	9.89	+0.41	14.31
Second	13.14	13.09	+0.05	16.91
Third	16.10	15.81	+0.29	21.21
Highest	21.05	21.15	$-0.10$	30.03

Table 2: Within-batch disagreement quartiles, pooled across four batches; 232 outcomes per row. Return errors and horizon-scaled volatility

V

are in pp. Rounded entries may not subtract exactly.

Figure 3: The raw disagreement–error gradient is also present for no change. A: absolute errors for the council and baseline. B: their paired difference within each quartile. Bars are pointwise asset-cluster 95% intervals.

Figure 4: Visible asset risk and council dispersion. A: 928 outcomes plotted on logarithmic axes. B: within-batch rank correlations. The council’s spread is closely related to historical volatility, which is more strongly related to absolute forecast error than spread is in each batch.

Figure 5: Adjusted and similar-risk comparisons. A: coefficients for standardized transformed predictors in the model of

\log(1+E)

, controlling for batch and asset class. B: lower and higher disagreement within five volatility groups per batch. Bars show pointwise asset-cluster 95% intervals. These are retrospective diagnostics.

Figure 6: Selective evaluation at equal within-batch coverage. A: absolute error among retained forecasts. B: paired error relative to no change. Points are observed estimates at 5% coverage increments; these curves have no uncertainty bands. Paired selector intervals are reported in the text. At 100% all rules retain the same panel. The rules require no fitted outcome labels; their evaluation is retrospective.

Figure 7: Two aggregation rules, two directional conclusions. A: mean-sign and modal-direction accuracy for the same agreement groups; modal ties fall back to mean sign. B: paired council-minus-no-change MAE, rather than visually inferred differences between raw-loss markers. Bars are pointwise asset-cluster 95% intervals, conditional on the observed periods.

Figure 8: Directional association varies across periods and aggregation rules. Each bar is high-minus-lower agreement accuracy within one batch (232 outcomes). These observed-period contrasts have no future-regime uncertainty interpretation.

Figure 9: Sensitivity of the high-minus-low disagreement contrast in excess error. Negative values indicate a smaller council-minus-baseline error in the highest-disagreement quartile. Pointwise asset-cluster 95% intervals span zero across the listed specifications.

Figure 10: Scale-aware diagnostics for the same quartiles. A: mean per-record error divided by historical horizon-scale volatility. B: pooled council skill relative to no change,

1-\sum E/\sum E^{0}

; positive favors the council. Points and bars are descriptive estimates without uncertainty bands.

Figure 11: Dependence among framework return forecasts across 928 outcomes. Pearson correlations pool four batches; the companion also supplies matrices within each batch. Neutral names map to historical archive labels in Appendix C . Correlation is not a measure of causal independence or model quality.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Batch	Council MAE	No-change MAE	Difference	Direction	Always up
3	18.88	16.97	+1.91	38.36%	52.16%
4	13.70	14.21	$-0.51$	50.43%	51.72%
5	13.47	14.08	$-0.62$	53.88%	59.91%
6	14.54	14.68	$-0.14$	50.86%	39.22%

Appendix

Table 3: Batch-level estimates in the primary panel, 232 outcomes per row. Return errors are in pp. Direction uses the three-category sign rule; always up is a separate directional baseline.

Disagreement quartile	Council RMSE (pp)	No-change RMSE (pp)
Lowest	17.69	16.55
Second	21.01	20.07
Third	23.39	22.79
Highest	31.14	30.63

Appendix

Table 4: Squared-error sensitivity. Both series exhibit increasing error across disagreement quartiles. This table supplies a second loss function, not a fitted probabilistic forecast evaluation.

Volatility window	Retained outcomes	Council MAE (pp)	Excess MAE (pp)
20 observations	232	7.60	+0.08 [ $-0.31$ , 0.45]
60 observations	232	7.97	$-0.03$ [ $-0.43$ , 0.37]
90 observations	232	7.50	+0.02 [ $-0.36$ , 0.41]

Appendix

Table 5: Lowest-volatility quarter under alternative historical windows. Brackets are pointwise asset-cluster 95% intervals for the paired error difference relative to no change.

Neutral name	Archive label	Interpretation
Paradigm shift	Elon Musk	Technology adoption, disruption, and execution constraints.
Contrarian forensic	Michael Burry	Downside asymmetry, balance-sheet fragility, and accounting risk.
Power audit	Niccolo Machiavelli	Incentives, control, counterparties, and distribution of economic rents.
Forensic deduction	Sherlock Holmes	Evidence chains, inconsistencies, and competing explanations.
Economic machine	Ray Dalio	Macro cycles, liquidity, debt, and regime transmission.
Civilizational compute	Superintelligence	Compute and energy constraints across long-term system transitions.

Appendix

Table 6: Explanatory framework map. Descriptions summarize configured perspectives, not historical human decisions.

Comparator	MAE (pp)	Excess (pp)	Direction
No change	14.99	0.00	2.48%
Council mean	15.15	0.16	48.38%
Council median	15.13	0.14	51.94%
Repeat prior 60-observation return	25.47	10.49	42.13%

Appendix

Table 7: Unfitted benchmark diagnostics on 928 outcomes. No-change direction is always flat under the three-category sign rule, not the always-up benchmark. Momentum is an intentionally simple comparator with a different lookback length.

Framework	MAE (pp)	Direction
Paradigm shift	15.76	49.78%
Contrarian forensic	17.01	47.84%
Power audit	15.64	51.72%
Economic machine	15.17	50.11%
Forensic deduction	15.93	47.95%
Civilizational compute	15.17	49.78%

Appendix

Table 8: Per-framework point forecasts, averaged across retained modes before scoring. These descriptive comparisons are not model-provider rankings.

Variable (pp unless noted)	Minimum	Median	95th pct.	Maximum
Absolute council forecast	0.01	3.16	11.35	22.24
Realized return	-52.10	0.83	40.57	205.78
Council dispersion	0.22	3.87	9.26	17.83
Horizon-scale historical risk	1.21	17.71	40.45	62.23
Horizon (calendar days)	71.00	88.00	90.00	91.00

Appendix

Table 9: Distribution summaries of the 928 primary records.

Subset	Records	Mean risk	Mean spread	Mean momentum	MAE
Primary balanced	928	20.61	4.41	4.34	15.15
Other valid-risk matched	293	27.78	5.54	7.92	17.00

Appendix

Table 10: Retained versus excluded valid-risk records; variables and MAE are in pp. The excluded set has 293 records across 141 assets, rather than four observations per asset. Differences describe selection and do not establish its cause.

Omitted batch	Mean-sign gap	Modal gap	Excess-error Q4–Q1
3	13.55	2.86	-0.59
4	7.07	-1.89	-0.35
5	6.56	-4.37	-0.59
6	9.41	-0.62	-0.53

Appendix

Table 11: Leave-one-batch-out point sensitivities. Accuracy gaps are high-minus-lower agreement in percentage points; excess-error gaps are pp. These overlapping subsets are not independent validation samples.

Batch	$D$ –risk	Risk–error	$D$ –error	$D$ –excess error
3	0.562	0.482	0.214	-0.026
4	0.641	0.420	0.286	-0.091
5	0.689	0.307	0.236	-0.025
6	0.595	0.371	0.321	0.009

Appendix

Table 12: Within-batch Spearman coefficients in the primary panel.

D

is numerical disagreement. These rank correlations are descriptive, not causally identified components of uncertainty.

Financial Forecasting

Financial forecasting uses market, company, and economic data to predict prices, returns, volatility, and business outcomes for decisions and risk management.

See all 89 papers

TimeSeek: Temporal Reliability of Agentic Forecasters
Apr 5, 2026Hamza Mostafa, Om Shastri, Dennis LeeLLM Agent Reliability Forecasting Benchmarks
The Wharton School, University of Pennsylvania
Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents
May 1, 2026Maksym Nechepurenko, Pavel ShuvalovAI Agent Evaluation Forecasting Benchmarks
Director of Research, Devnull, Dubai, UAE · Chief Technology Officer, Devnull, Dubai, UAE
Same Person, Different Advice: Auditing LLM Financial Advice with Repeated Risk-Tolerance Responses
Oct 16, 2026Soovadeep Bakshi, Apurba Bose, Aishwarya Bose +1LLM Auditing LLM Decision-Making
Independent Researcher

Apr 5, 2026J/K move · Enter open · S save

TimeSeek: Temporal Reliability of Agentic Forecasters

Hamza Mostafa, Om Shastri, Dennis Lee

The Wharton School, University of Pennsylvania

LLM Agent Reliability Forecasting Benchmarks Probabilistic Forecasting Prediction Markets

Open paper AskPDF arXiv

We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.

Abstract

Financial Forecasting

Explore similar work

TimeSeek: Temporal Reliability of Agentic Forecasters

Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

Same Person, Different Advice: Auditing LLM Financial Advice with Repeated Risk-Tolerance Responses