The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?

Xaver Raphael Davey; Shubhra Mishra; Anton Adelöw; Siyao Liu

cs.CLOct 12, 2026

The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?

Authors: Xaver Raphael Davey, Shubhra Mishra, Anton Adelöw, Siyao Liu

Organizations: KTH Royal Institute of Technology

Abstract

Text generated by Large Language Models (LLMs) often remains limited in diversity compared to that produced by humans. Recent advances demonstrate the presence of inter- and intra-model collapse in open-ended generation of English text, raising concerns about the long-term homogenization of LLM, and more importantly, human thought. However, our understanding of whether this homogenization extends to languages beyond English is limited. In this work, we investigate the role of language on output diversity. Specifically, we aim to understand whether multilingual LLM-generated text is also homogeneous, and whether this homogeneity matches what we observe in humans. Across 23 open-weight models and 13 languages, we find that the artificial hivemind effect is pervasive but uneven in its extent. The intra-model effect holds in three different embedding spaces and under a lexical metric that uses no embedding model, so it is not an artifact of how diversity is measured. The size of the homogenization gap is associated with a model's competence in a language, measured on a held-out translation benchmark, robustly across embedding spaces. Thus, we demonstrate a performance-diversity tradeoff in multilingual LLMs.

Figures & tables

Figure 1: Per-language Principal Component Analysis (PCA) projections of response embeddings. For four representative languages (English, Japanese, Basque, and Chinese), blue point clouds represent the spatial distribution of pooled LLM generations, while red $\times$ markers display the human answers to the same prompts. LLM generations concentrate into dense modes, whereas the human answers are spread thinly over the same region. Each panel pools all 30 prompts of its language, so it shows the mode structure of the pooled distribution rather than the within-prompt gap measured in Section 4 . Each of the two plotted components explains 5–10% of the embedding variance in its panel.

Panel of Figure 2 — Figure 2: Cross-respondent cosine similarities. $LLM\leftrightarrow LLM$ vs. $Human\leftrightarrow Human$ evaluated (a) globally and (b) per individual model, and broken down (c) per language and (d) per model-family pair, on identical prompts. Higher LLM bars signify tighter cross-model clustering than humans exhibit with each other; in (b) and (d) the human baseline does not vary along the axis, so it appears once as the first bar, with a dashed reference line at its height. Whiskers are standard errors over prompts; stars mark Holm-corrected Wilcoxon signed-rank tests on the per-prompt differences (Appendix D ): ${}^{*}p<0.05$ , ${}^{**}p<0.01$ , ${}^{***}p<0.001$ .

Figure 3: Homogenization gap for each language $\times$ model pair. Red tiles indicate high levels of homogenization compared to humans, while blue tiles indicate higher generative diversity. Stars as in Figure 2 , corrected across the 299 cells.

Figure 4: Respondent–respondent similarity, aggregated by model family. A respondent is any source of an answer to a prompt: one of the 23 LLMs, or the pooled human annotators. Each cell is the mean pairwise cosine similarity between answers from two different respondents to the same prompt, averaged over all prompts and languages ( $\times 100$ ).

Similarity metric	Same model	Diff. humans	Gap	Langs.	$\rho$
text-embedding-3-small	0.857	0.657	+0.200	13/13	—
BGE-M3	0.870	0.727	+0.143	13/13	0.92
BGE-M3, 500-char cap	0.857	0.723	+0.134	13/13	0.91
Qwen3-Embedding-0.6B	0.866	0.666	+0.200	13/13	0.93
char 4-gram Jaccard	0.275	0.133	+0.141	13/13	0.71
char 4-gram Jaccard, 500-char cap	0.268	0.130	+0.138	13/13	0.72

Table 1: The homogenization gap under three embedding spaces and an embedder-free lexical metric. Mean similarity between two answers to the same prompt when they come from the same model resampled and from two different humans, averaged over all prompts and models, and the difference between them (the homogenization gap). Langs. counts languages with a positive gap;

\rho

is the Spearman correlation of the per-cell gap against the primary metric. The effect is present in every language under every metric. The BGE-M3 500-char row re-embeds every response cut to its first 500 characters, on 10 of the 50 samples per prompt.

Panel of Figure 5 — Figure 5: Linguistic predictors of the homogenization gap. (a) One point per (model, language) cell, the level the effect operates at; averaged per language it vanishes ( $\rho=0.06$ , $p=0.85$ ), since the effect lives across models within a language (specification H). (b) Fitted separately in each embedding space. Competence predicts the gap in all three spaces; resource level, fertility and morphology in none.

Panel of Figure 6 — Figure 6: LLM versus Human intra-similarity. (a) Pooled globally, (b) per model family, (c) per language and (d) per individual model. Lower values signify higher generative diversity. Whiskers are standard errors over prompts, and stars mark Holm-corrected Wilcoxon signed-rank tests on the per-prompt differences (Appendix D ): ${}^{*}p<0.05$ , ${}^{**}p<0.01$ , ${}^{***}p<0.001$ .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Checkpoint	Rev.	Prec.	GPUs	Chars	Trunc. %	Think
Qwen3 0.6B	Qwen/Qwen3-0.6B	c1899de2	bf16	1 $\times$ A40 48 GB	2,320	1.1	yes
Qwen3 1.7B	Qwen/Qwen3-1.7B	70d244cc	bf16	1 $\times$ A40 48 GB	4,442	14.8	yes
Qwen3 4B	Qwen/Qwen3-4B-Instruct-2507	cdbee75f	bf16	1 $\times$ A40 48 GB	2,546	7.2	no
Qwen3 8B	Qwen/Qwen3-8B	b968826d	bf16	1 $\times$ A40 48 GB	4,635	18.5	yes
Qwen3 14B	Qwen/Qwen3-14B	40c06982	bf16	1 $\times$ A40 48 GB	4,175	12.4	yes
Qwen3 32B	Qwen/Qwen3-32B	9216db57	bf16	2 $\times$ A100 80 GB	4,547	17.2	yes

Appendix

Table 2: Breakdown of the 23 open-weight model variants across 5 families utilized in the multilingual homogeneity evaluation. Revision is the Hugging Face commit that was the repository head when generation ran. Precision is the dtype declared in the checkpoint’s config. Chars is the mean response length in characters, reasoning block included where present; Trunc. is the share of responses cut off by the 2048-token limit; Think marks checkpoints whose responses carry a <think> reasoning block.

Figure 7: Per-language PCA projections of response embeddings for all 13 languages. Blue point clouds are pooled LLM generations; red $\times$ markers are human answers. Each of the two plotted components explains 5–10% of the embedding variance in its panel.

Figure 8: Inter-model gap across family-pair combinations $\times$ languages. Positive values (red blocks) indicate that the chosen model pairs cluster more tightly than human responses within that language. Stars mark Holm-corrected Wilcoxon signed-rank tests across the 195 cells: ${}^{*}p<0.05$ , ${}^{**}p<0.01$ , ${}^{***}p<0.001$ .

Lang.	BGE-M3	BGE-M3, 500-char cap	Qwen3-Emb	OpenAI	Jaccard	Jacc. 500c
en	$+0.156$	$+0.144$	$+0.203$	$+0.251$	$+0.174$	$+0.165$
de	$+0.150$	$+0.136$	$+0.203$	$+0.180$	$+0.177$	$+0.173$
fr	$+0.165$	$+0.155$	$+0.212$	$+0.231$	$+0.149$	$+0.148$
es	$+0.155$	$+0.145$	$+0.230$	$+0.225$	$+0.182$	$+0.172$
it	$+0.158$	$+0.146$	$+0.191$	$+0.187$	$+0.166$	$+0.161$
pt-BR	$+0.099$	$+0.084$	$+0.168$	$+0.147$	$+0.140$	$+0.122$

Appendix

Table 3: Per-language homogenization gap (LLM minus human intra-similarity) under each similarity metric, averaged over the 23 models. Every language shows a positive gap under every metric.

Figure 9: Respondent–respondent similarity for all 24 respondents. We report for each of the 23 LLMs, grouped by family, and the pooled human annotators (boxed), the mean pairwise cosine similarity between answers to the same prompt, averaged over all prompts and languages ( $\times 100$ ). Off-diagonal cells compare two different respondents; the LLM diagonal resamples one model at $T=1.0$ . Of the 253 distinct LLM–LLM pairs, 245 exceed the human–human value of 65.7; the eight that do not all involve Gemma-3 270M or Qwen3 0.6B, whose non-English output degenerates.

Metric	Comparison / level	$k$	gap $>0$	CI $>0$	perm. Holm	Wilc. Holm	perm. BH
text-embedding-3-small	Intra-model: per family	5	5	5	5	5	5
	Intra-model: per model	23	23	23	23	23	23
	Intra-model: per language	13	13	13	13	13	13
	Intra-model: per (model, language)	299	290	281	271	270	280
	Inter-model: per language	13	12	12	12	12	12
	Inter-model: per model	23	23	22	22	22	22

Appendix

Table 4: Paired prompt-level tests of the homogenization gap, with multiple-comparison correction. Each group is tested on one LLM-minus-human difference per prompt (

n=390

prompts for per-model and per-family-pair groups,

30

otherwise). Columns count the groups at that level (

k

) whose mean gap is positive, whose 95% prompt-bootstrap interval lies above zero, and that are positive and significant at

0.05

after Holm correction of the sign-flip permutation and Wilcoxon signed-rank

p

-values, and after Benjamini–Hochberg correction of the permutation

p

-values, each corrected across the

k

groups of its row (a group that is significant in the reverse direction, such as Basque in the inter-model comparison, is not counted). The last row of the first block tests the same-model versus different-model comparison of Section 4.1 , one model resampled against two different models, which needs no human baseline. The alternative metrics are reported for the intra-model comparison.

Family	en	de	fr	es	it	pt-BR	ca	pl	ru	vi	zh	ja	eu
Aya Expanse	1.00	1.32	1.35	1.26	1.32	1.16	1.58	1.34	1.27	1.33	1.04	1.22	1.76
Gemma-3	1.00	1.33	1.41	1.30	1.38	1.26	1.54	1.51	1.37	1.37	1.08	1.21	1.68
Mistral	1.00	1.40	1.42	1.41	1.45	1.37	1.47	1.65	1.24	1.84	1.12	1.33	1.58
OLMo-2	1.00	1.58	1.63	1.54	1.64	1.48	1.72	1.91	2.45	2.43	1.86	2.25	1.88
Qwen3	1.00	1.56	1.58	1.52	1.63	1.45	1.69	1.76	1.75	1.44	1.00	1.40	1.87

Appendix

Table 5: Tokenization fertility on FLORES-200 devtest. We report subword tokens each family’s tokenizer spends per parallel sentence, relative to its own English count. Basque and Polish cost the most; Chinese the least.

Space	Spec	Competence	Resource level	Tokenization fertility	Morph. complexity	GlotLID
OpenAI	A	$+0.036^{***}$	$+0.023$	$-0.001$	$+0.015$	—
	B	$+0.038^{***}$	$+0.020$	$-0.006$	—	—
	C	$+0.044^{***}$	$+0.011$	$-0.004$	—	—
	F	$+0.023^{*}$	$+0.019$	$-0.002$	—	$+0.024^{***}$
BGE-M3	A	$+0.038^{***}$	$+0.015$	$+0.000$	$+0.018$	—
	B	$+0.040^{***}$	$+0.013$	$-0.002$	—	—

Appendix

Table 6: Standardised coefficients from the crossed-effects model of the homogenization gap, fitted in each embedding space. Competence is FLORES+ English

\to

target translation chrF++, a held-out benchmark that shares no prompts or outputs with the similarity scores; English is excluded, since the translations are from English. Spec A uses all four predictors on the languages for which morphology is defined (210 cells; the type-token ratio is undefined for zh / ja ); B drops morphology to recover them (252 cells); C repeats B on the 230 cells whose generations have a GlotLID target-language match rate

\geq 0.80

, so degenerate output cannot drive the fit; F adds that match rate to B as a fifth predictor, testing whether it carries signal beyond competence. Competence is positive and significant (

p\leq 0.003

) in every specification and embedding space, including F, where the GlotLID match rate is also significant. Resource level, tokenization fertility and morphological complexity are significant in none.

{}^{*}p<0.05

{}^{***}p<0.001

Output Diversity in Language Models

See all 32 papers

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Oct 27, 2025Liwei Jiang, Yuanjun Chai, Margaret Li +7Open-Ended Generation LLM Evaluation
University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University +3
How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Models
Sep 27, 2026Rylan Schaeffer, Brando Miranda, Joshua Kazdan +2Output Diversity in Language Models Homogenization in Language Models
Stanford Computer Science
Multilingual Large Language Models do not comprehend all natural languages to equal degrees
Feb 23, 2026Natalia Moskvina, Raquel Montero, Masaya Yoshida +8Multilingual Language Models Multilingual Language Model Evaluation
Universitat Autònoma de Barcelona · Institució Catalana de Recerca i Estudis Avançats (ICREA) · Radboud University Nijmegen +1

Oct 27, 2025J/K move · Enter open · S save

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu+6

University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University+3

Open-Ended Generation LLM Evaluation Language Model Generation Evaluation Diverse Text Generation Mode Collapse Output Diversity in Language Models

Open paper AskPDF arXiv

Language models (LMs) often struggle to generate diverse, human-like creative content, raising concerns about the long-term homogenization of human thought through repeated exposure to similar outputs. Yet scalable methods for evaluating LM output diversity remain limited, especially beyond narrow tasks such as random number or name generation, or beyond repeated sampling from a single model. We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries that admit a wide range of plausible answers with no single ground truth. We introduce the first comprehensive taxonomy for characterizing the full spectrum of open-ended prompts posed to LMs, comprising 6 top-level categories (e.g., brainstorm & ideation) that further breaks down to 17 subcategories. Using Infinity-Chat, we present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect in open-ended generation of LMs, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and more so (2) inter-model homogeneity, where different models produce strikingly similar outputs. Infinity-Chat also includes 31,250 human annotations, across absolute ratings and pairwise preferences, with 25 independent human annotations per example. This enables studying collective and individual-specific human preferences in response to open-ended queries. Our findings show that LMs, reward models, and LM judges are less well calibrated to human ratings on model generations that elicit differing idiosyncratic annotator preferences, despite maintaining comparable overall quality. Overall, INFINITY-CHAT presents the first large-scale resource for systematically studying real-world open-ended queries to LMs, revealing critical insights to guide future research for mitigating long-term AI safety risks posed by the Artificial Hivemind.

Abstract

Output Diversity in Language Models

Explore similar work

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Models

Multilingual Large Language Models do not comprehend all natural languages to equal degrees