The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission

Theodor Spiro

cs.CYApr 7, 2026

The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission

Authors: Theodor Spiro

Abstract

When large language models (LLMs) are consulted as forecasting tools, the independence of individual errors -- the foundation of collective intelligence -- may collapse. We test three conditions necessary for this "epistemic monoculture" to emerge. In Study 1, we show that GPT-4o, Claude, and Gemini exhibit highly correlated forecasting errors on 568 resolved binary prediction questions (mean pairwise error correlation r = 0.77, p < 0.001; r = 0.78 excluding likely-leaked questions), despite being developed independently by different organizations. In Study 2, we test whether this correlated bias has propagated into human crowd forecasts, using a within-question design that tracks community prediction shifts across the ChatGPT launch boundary (November 2022). We find that community forecasts move in the direction predicted by LLMs (r = 0.20, p = 0.007), but this shift is fully explained by rational updating toward ground truth. In Study 3, we examine whether the category-level pattern of human forecasting errors increasingly resembles the LLM bias fingerprint. We find the opposite: pre-ChatGPT human biases already strongly resembled the LLM pattern (r = 0.87), while post-ChatGPT the resemblance weakened (r = -0.28). Together, these findings reveal an epistemic monoculture that is built but not yet activated: three nominally independent AI systems share the same failure modes, amplifying precisely the biases humans already hold.

Figures & tables

Metric	GPT-4o	Claude	Gemini	Community
Brier score (full, $N{=}568$ )	0.183	0.161	0.182	0.084
Brier score (clean, $N{=}414$ )	0.242	0.212	0.248	0.100
Directional bias	$+0.118$	$+0.071$	$+0.140$	—
ECE	0.115	0.079	0.149	—

Table 1: Individual model performance on resolved binary questions.

Model Pair	Error Correlation ( $r$ )	$p$ -value
GPT-4o $\times$ Claude	0.822	$<0.001$
GPT-4o $\times$ Gemini	0.748	$<0.001$
Claude $\times$ Gemini	0.742	$<0.001$
Mean	0.771

Table 2: Pairwise error correlations between LLM predictions (

N=568

Figure 1: Pairwise error scatter plots for three LLMs on 568 resolved binary forecasting questions. Each point represents one question; axes show signed prediction error (

p-o

) for each model. High correlation (

r=0.74

–

0.82

) indicates that models fail on the same questions in the same direction. Banding at extreme values (0 and 1) is consistent with training data leakage for a subset of questions.

Predictor	$\beta$	SE	$t$	$p$
Intercept ( $\beta_{0}$ )	$-0.015$	0.004	$-3.44$	$<0.001$
Resolution pull ( $\beta_{1}$ )	$+0.061$	0.014	4.25	$<0.001$
LLM pull ( $\beta_{2}$ )	$+0.023$	0.025	0.92	0.36

Table 3: Regression results: Predictors of community prediction shift (

N=179

Figure 2: Calibration curves for three LLMs and the Metaculus community prediction. Predicted probabilities are binned into deciles; observed frequencies are plotted against bin midpoints. The diagonal line represents perfect calibration. All three LLMs show overconfidence in the mid-range (0.3–0.7), while the community prediction tracks the diagonal closely.

Period	Fingerprint Correlation ( $r$ )	$p$	$N$ categories
Pre-ChatGPT	0.874	0.053	5
Post-ChatGPT	$-0.275$	0.654	5

Table 4: Fingerprint correlation: LLM bias pattern vs. human bias pattern by period.

Category	$N$ (pre/post)	LLM bias	Pre-ChatGPT human bias	Post-ChatGPT human bias
Economics	20 / 11	$+0.079$	$+0.001$	$+0.017$
Geopolitics	90 / 19	$+0.074$	$+0.068$	$-0.065$
Health	34 / 6	$+0.121$	$-0.016$	$+0.168$
Society	31 / 15	$+0.167$	$+0.133$	$+0.027$
Technology	16 / 8	$+0.249$	$+0.250$	$-0.081$

Table 5: Category-level bias profiles.

Figure 3: Bias fingerprint comparison across five question categories. Left: LLM bias (mean across three models) and pre-ChatGPT human bias by category, showing strong correspondence (

r=0.87

). Right: Post-ChatGPT human bias diverges from the LLM pattern (

r=-0.28

), suggesting that the fingerprint weakened rather than strengthened after LLMs became widely available.

Correlated Errors in Language Models

See all 8 papers

Diversity is the Strength of the AI Crowd
Jun 29, 2026Matthew Aitchison, Scott Jeen, Toby Shevlane +1Forecasting Benchmarks Language Model Ensembles
1Mantic Technologies, London, UK.
Predicting Biased Human Decision-Making with Large Language Models in Conversational Settings
Jan 16, 2026Stephen Pilli, Vivek NallurLLM Decision-Making Human-AI Decision Making
University College Dublin · Dublin, Ireland
Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
May 26, 2026Hui Dai, Ryan Teehan, Parsa Torabian +1Post-Hoc Calibration Probabilistic Forecasting
Agentic Learning AI Lab, New York University · Chronologies AI

Jun 29, 2026J/K move · Enter open · S save

Diversity is the Strength of the AI Crowd

Matthew Aitchison, Scott Jeen, Toby Shevlane, Ben Day

1Mantic Technologies, London, UK.

Forecasting Benchmarks Language Model Ensembles Ensemble Diversity

Open paper AskPDF arXiv

Top AI forecasting systems are approaching superforecaster-level accuracy on future world events, but still rely primarily on off-the-shelf LLMs combined with forecasting-specific context gathering and scaffolding. We study how to improve this recipe through ensembling: given a fixed number of samples, which off-the-shelf model forecasts should be combined to maximize accuracy? On binary questions from the Metaculus AI Benchmark, we find that individual accuracy is not enough: many frontier LLMs make highly correlated predictions, limiting the value of additional forecasts from the same or similar models. Instead, the strongest ensembles combine accurate but diverse forecasters, with models such as \model{Grok 4} contributing disproportionately because their predictions are less correlated with other frontier LLMs. These results suggest that the strength of the AI crowd comes not from sampling more forecasts indiscriminately, but from combining forecasts across models with complementary errors, motivating forecasting systems that explicitly optimize for both model quality and diversity.

Abstract

Correlated Errors in Language Models

Explore similar work

Diversity is the Strength of the AI Crowd

Predicting Biased Human Decision-Making with Large Language Models in Conversational Settings

Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting