Diverse by Reasoning: Harnessing the Wisdom of LLM Crowds for Future Prediction

Nirupam Chetlapalli; Yiming Liao; Min-Chun Chen; Keke Chen

cs.AIAug 25, 2026

Diverse by Reasoning: Harnessing the Wisdom of LLM Crowds for Future Prediction

Authors: Nirupam Chetlapalli, Yiming Liao, Min-Chun Chen, Keke Chen

Organizations: Computer Science and Electrical Engineering University of Maryland, Baltimore County Baltimore, MD, USA

Abstract

Large language models (LLMs) are increasingly used for future prediction, motivating the use of multiple models as a wisdom-of-the-crowd mechanism. However, simply increasing crowd size does not guarantee effective diversity, as different LLMs may exhibit redundant behaviors. We propose a behavior-aware framework for constructing diverse LLM crowds. The framework characterizes models using their reasoning traces on independent development tasks, clusters models by behavioral similarity, and selects representatives for collective prediction. We evaluate 25 LLMs using seven development benchmarks for behavioral diversity modeling and two future-prediction benchmarks for evaluating diverse crowds' performance. Our results show that crowd composition can matter more than crowd size: a three-model medoid crowd based on K-means++ behavioral clustering outperforms conventional voting over all 25 models on both prediction benchmarks, while reducing model calls by 88% and inference cost by approximately 80%. The results further suggest that representative behavioral diversity, rather than simply maximizing diversity, is important for constructing effective LLM crowds

Figures & tables

Model	Weights	Cutoff / Release	Model	Weights	Cutoff / Release
OpenAI			DeepSeek
gpt-5.4	Closed	2025–08–31	deepseek-chat-v3.1	Open	2025–03–31
gpt-5.4-mini	Closed	2025–08–31	deepseek-v3.2	Open	2025–12–01
Anthropic			Moonshot AI
claude-haiku-4.5	Closed	2025–10	kimi-k2.5	Open	2026–01–27
claude-sonnet-4.6	Closed	2026–01	kimi-k2.6	Open	2025–04

TABLE I: The

N=25

LLMs in the experimental crowd, grouped by developer. Cutoff / Release is the training cutoff where the provider publishes one (16 of the 25 models), and the release date otherwise (9). A model cannot be trained on data that did not yet exist when it was released, so for those 9 the real cutoff can only be earlier than the date shown. The latest date across the crowd is 2026–02–16.

Benchmark	# Questions	Capability	Resolves
LiveBench/Reasoning	50	General reasoning	—
LiveBench/Math	50	Mathematical reasoning	—
LiveBench/Instruction-Following	50	Instruction following	—
LiveBench/Data-Analysis	50	Data analysis	—
GPQA Diamond	50	Scientific reasoning	—
Natural Plan	50	Planning	—

TABLE II: Benchmarks used for behavioral characterization (top 7 rows) and future-prediction evaluation. Resolves is the range of dates on which the questions’ outcomes became known, for direct comparison with the training cutoffs in Table I : the earliest of these, 2026–03–07, falls after the latest cutoff in the crowd, 2026–02–16. The development benchmarks are not time-indexed and have no resolution date.

Panel of Figure 3 — Fig. 1: Silhouette score of $K$ -means++ across different numbers of clusters. While $K=2$ has the highest SC, it may lead to many ties for medoid voting. Thus, $K=3$ is selected as a more practical choice.

Panel of Figure 4 — Fig. 2: Both panels place the 25 models at identical t-SNE coordinates; Larger marks are medoids. Note that the hierarchical clustering result is unevenly sized: grok-4.3 is the only member of a cluster. The k-means++ results are stable with a stability score of 0.767.

Panel of Figure 5 — Fig. 3: Score against the number of representatives drawn from each cluster. Colour is the cluster count $K$ , line style the clustering algorithm (solid HC = hierarchical, dashed KM = $K$ -means++); a label of the form HC = KM marks two clusterings that produce identical committees at every $k$ and are therefore drawn once. $k{=}1$ is the cluster medoid; $k\geq 2$ takes the $k$ most mutually dissimilar members, and all representatives are pooled into a single vote. The grey dashed line is all-model voting over the full 25-model crowd. Scores are over 100 held-out items from a seeded five-fold split. Clusterings with no dissimilar set defined beyond $k{=}2$ stop there.

Method	FutureX-Past	BTF-v3	Avg. Rank	# Models
Performance-based expert	0.265 (10)	0.790 (7)	8.5	1
All-model voting	0.296 (2)	0.800 (4)	3.0	25
Random representatives (KM)	0.274 (6)	0.771 (10)	8.0	3
Random representatives (HC)	0.285 (3)	0.798 (6)	4.5	3
Medoid voting (KM)	0.302 (1)	0.810 (2)	1.5	3
Medoid voting (HC)	0.270 (8)	0.830 (1)	4.5	3

TABLE III: Future-prediction performance of individual and crowd-based methods, over 100 held-out items from a seeded five-fold split, pooled so that every question contributes exactly once. Per-benchmark rank in parentheses; Avg. Rank is the mean of the two ranks rather than of the scores, because the two metrics are on different scales. HC = hierarchical clustering, KM =

K

-means++, both at

K=3

. The best value in each benchmark column is shown in bold.

		FutureX-Past		BTF-v3
Method	Calls	USD (rel.)	Score	USD (rel.)	Score
Performance-based expert	100	0.11 (0.01)	0.265	1.31 (0.06)	0.790
All-model voting	2500	10.96 (1.00)	0.296	21.26 (1.00)	0.800
Random representatives	300	1.25 (0.11)	0.274	2.41 (0.11)	0.771
Medoid voting	300	2.10 (0.19)	0.302	4.51 (0.21)	0.810
$k=4$ dissimilar voting	1200	7.12 (0.65)	0.272	13.91 (0.65)	0.780

TABLE IV: Cost of each method per 100 questions, against the score it achieves. Calls are the queries a method commits to,

m

models

\times

100 questions, whether or not every one returns a usable answer. Dollars are the amounts the provider actually charged, not a price-times-tokens estimate. Rel. is cost relative to all-model voting on the same benchmark.

K

-means++ clustering,

K=3

Panel of Figure 8 — Fig. 4: Future-prediction score against measured cost in USD, for the $K$ -means++ clustering at $K=3$ . Medoid voting sits above and to the left of all-model voting on both benchmarks: a higher score for about 20% of the outlay. Costs are the amounts the provider actually charged, summed over the 100 questions of each benchmark.

Panel of Figure 9 — Fig. 5: Future-prediction score against the number of LLM calls, for the $K$ -means++ clustering at $K=3$ . Medoid voting sits above and to the left of all-model voting on both benchmarks: a higher score for about 12% of the calls. A method’s call count is the queries it commits to, $m$ models $\times$ 100 questions, whether or not every one returns a usable answer.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Stage 1	Stage 2	Wins	$\Delta$ vs random	$p$
defer, split weight	reverse alphabetical	6/6	$+0.0192$	0.191
defer, full weight	reverse alphabetical	6/6	$+0.0176$	0.218
cluster medoid	reverse alphabetical	5/6	$+0.0146$	0.245
reverse alphabetical	cluster population	6/6	$+0.0121$	0.275
reverse alphabetical	alphabetical	5/6	$+0.0116$	0.283
alphabetical	alphabetical	0/6	$-0.0222$	0.757

Appendix

TABLE V: Joint tie-break strategies on FutureX-Past, scored against RANDOM resolution at both stages.

\Delta

is the mean change in the tier-weighted score across the 6 two-stage committees (

K=3

k\in\{2,3,4\}

, both clusterings); wins counts the committees on which the strategy scores above random.

p

is measured against 300 ARBITRARY deterministic rule pairs, which is the luck available to a convention that carries no information about correctness. Random is the reference and is therefore exactly zero. No strategy separates from the arbitrary-rule null.

Language Model Ensembles

Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.

See all 137 papers

Diversity is the Strength of the AI Crowd
Jun 29, 2026Matthew Aitchison, Scott Jeen, Toby Shevlane +1Forecasting Benchmarks Language Model Ensembles
1Mantic Technologies, London, UK.
Is the Future Predictable? Modeling Future Predictability for Reliable Temporal Reasoning
Oct 8, 2026Yongyi Huang, Huaping Zhang, Wenyao Cui +3Temporal Reasoning in Language Models LLM Reliability
Beijing Institute of Technology · Zhongguancun Academy
Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts
Sep 27, 2026Cheng Peng, Ruixi Luo, Zhi Chen +1Language Model Ensembles Probabilistic Forecasting
Chinese University of Hong Kong

Jun 29, 2026J/K move · Enter open · S save

Diversity is the Strength of the AI Crowd

Matthew Aitchison, Scott Jeen, Toby Shevlane, Ben Day

1Mantic Technologies, London, UK.

Forecasting Benchmarks Language Model Ensembles Ensemble Diversity

Open paper AskPDF arXiv

Top AI forecasting systems are approaching superforecaster-level accuracy on future world events, but still rely primarily on off-the-shelf LLMs combined with forecasting-specific context gathering and scaffolding. We study how to improve this recipe through ensembling: given a fixed number of samples, which off-the-shelf model forecasts should be combined to maximize accuracy? On binary questions from the Metaculus AI Benchmark, we find that individual accuracy is not enough: many frontier LLMs make highly correlated predictions, limiting the value of additional forecasts from the same or similar models. Instead, the strongest ensembles combine accurate but diverse forecasters, with models such as \model{Grok 4} contributing disproportionately because their predictions are less correlated with other frontier LLMs. These results suggest that the strength of the AI crowd comes not from sampling more forecasts indiscriminately, but from combining forecasts across models with complementary errors, motivating forecasting systems that explicitly optimize for both model quality and diversity.

Abstract

Language Model Ensembles

Explore similar work

Diversity is the Strength of the AI Crowd

Is the Future Predictable? Modeling Future Predictability for Reliable Temporal Reasoning

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts