How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Models

Rylan Schaeffer; Brando Miranda; Joshua Kazdan; Jessica Chudnovsky; Sanmi Koyejo

cs.LGSep 27, 2026

How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Models

Authors: Rylan Schaeffer, Brando Miranda, Joshua Kazdan, Jessica Chudnovsky, Sanmi Koyejo

Organizations: Stanford Computer Science

Abstract

Recent research argues that language models exhibit pronounced homogeneity in open-ended generation, framing such behavior as an Artificial Hivemind that poses a long-term threat to human creativity. We examine three of its central results. First, the flagship example is that model responses to "Write a metaphor involving time" collapse into two clusters. Visualization, spectral analysis, clustering, and language model labels all contradict this description. The labels record each response's vehicle, what it compares time to. Our responses and the original authors' own show one dominant vehicle plus a heavy tail of distinct minority vehicles. "Time" is one of our least diverse topics, so the example is a favorable case, not a representative one. Second, the paper measures homogeneity against an undemanding null: responses to unrelated prompts. Under a more demanding null (same-prompt responses expressing genuinely different ideas), 20%-32% of such pairs already exceed the paper's 0.8 convergence threshold. A residual effect survives this null. The paper's same-prompt pairs exceed 0.8 roughly two to three times as often as our different-idea pairs. Much of what the paper calls homogeneity is the shared geometry of answering the same prompt. The remaining measurements lack any null: no human baseline is collected, and the model-indistinguishability statistic has no null. Third, the paper concludes that inference-time interventions are inadequate for combating the Artificial Hivemind, writing that "more generalizable solutions are needed at the model training level." We show that this conclusion is unsupported in three ways, and that an inference-time intervention (prompting) reliably raises measured response diversity. We do not resolve whether the Artificial Hivemind is real. We show that the published evidence does not establish it.

Figures & tables

Figure 1: The Artificial Hivemind’s Flagship Example Is Undercut by Five Lines of Evidence. Figure 1 from Jiang et al. (2025) . When 25 language models are each prompted 50 times: “Write a metaphor involving time,” the authors found: “Despite the diversity of model families and sizes, the responses form just two primary clusters”. Our reproduction and re-analysis of the original data undercuts this description (Section 2 ).

Figure 2: Across 12 Topics in Our Reproduction, Every Topic Shows More Than Two Clusters. PCA projections of embedded responses for each of the 12 baseline topics “Write a metaphor about {topic}”, reproducing the original flagship figure visualization. 750 responses per panel: 15 models $\times$ 50 samples. We selected 12 topics: “time” to match the original paper, plus 11 others to test the existence of the Hivemind and how representative “time” is. Every panel shows more than two clusters.

Figure 3: The Eigenspectra Show No Sharp Elbow, and Two Principal Components Fail to Capture the Majority of the Variance. Cumulative variance explained versus number of principal components, for all 12 topics under the prompt “Write a metaphor about {topic}”. These eigenspectra show that (1) no sharp elbow exists, contrary to what the tight-two-cluster reading of the flagship figure predicts, and (2) the first two dimensions are not representative. 2 components capture only 25–41% of variance, depending on topic and embedding model; reaching 90% of variance requires 60–110 components. The 2D PCA visualizations are not a summary; they are a lossy compression that discards most of the structure.

Figure 4: Every Clustering Method Selects More Than Two Clusters. Each method’s selected number of clusters $K$ over the 12 baseline topics. One small marker per topic; the diamond is the median over topics. The dashed line marks $K{=}2$ , the original paper’s claim. The methods span probabilistic parametric, nonparametric density-based, and graph-based approaches. Per-method ranges (Appendix D ): vMF 128-dim $22$ – $41$ , vMF 256-dim $20$ – $52$ , HDBSCAN $6$ – $17$ , spectral eigengap $4$ – $23$ . Fig. 9 (appendix) gives the per-topic values.

Figure 5: On the Original Data, BIC Selects More Than Two Clusters. vMF-mixture BIC versus

K

on Gemma Matryoshka slices of Jiang et al. (2025) ’s model responses (privately shared with us) to “Write a metaphor involving time.” (

n=2{,}150

43

models

\times

50

samples), same procedure as Fig. 4 : BIC selects

K^{*}{=}42

(128-dim) and

K^{*}{=}44

(256-dim) and never selects

K{=}2

or anything near it. HDBSCAN ( min_cluster_size

=15

, min_samples

=1

) finds

K{=}5

(10.7% noise) and the spectral eigengap (

k

-NN,

k{=}20

) suggests

K{=}4

: coarser than on our data, but still not two. See Appendix F for more information.

Vehicle	$n$ responses	Share of all $750$
river	604	80.5%
sculptor	138	18.4%
pickpocket	123	16.4%
weaver	102	13.6%
fire	45	6.0%
currency	31	4.1%

Table 1: Language Model Labels Suggest One Dominant Vehicle Plus a Heavy Tail, Not the Two Clusters. Distribution of our

750

responses to “Write a metaphor about time” (

15

models

\times

50

samples) over vehicles a language model assigned from the response text alone, with no embedding (procedure in Appendix G ). Vehicles carried by fewer than

7

responses (0.93% of the pool) are pooled. A response counts once per vehicle it carries, so shares do not sum to

1

. The effective number of vehicles is computed over the

1{,}229

vehicle–response marks, not over the response column. Those

1{,}229

are the

1{,}173

in the

14

kept rows plus

56

in the tail. A response that offers several comparisons can carry more than one tail vehicle.

11

of the

45

tail responses carry two, so the tail holds

56

vehicle–response marks over

45

responses. Tables 6 and 7 in Appendix G give the menu-answering caveat and the committed-only restriction.

Null distribution	Median	Gap to 0.8 threshold
Different prompts ( Jiang et al., 2025 )	${\sim}0.11$	${\sim}0.69$ (99.95th pctl)
Same prompt, genuinely different ideas (ours):
geometric clustering, 50 prompts	$0.723$	${\sim}0.08$
LLM-labeled, 15 prompts	$0.703$	${\sim}0.10$

Table 2: The Hivemind’s Convergence Threshold Nearly Coincides with the Floor Set by Genuinely Different Ideas. Median cosine similarity under text-embedding-3-small for the paper’s null (responses to different prompts) versus our two constructions of a more demanding null (same-prompt responses expressing genuinely different ideas, determined using either geometric clustering or LLM labeling).

Figure 6: An Unsupported Premise: The Min-

p

Authors’ Own Human Evaluation Indicates Min-

p

Sampling Does Not Increase Diversity. The visualization is adapted from Fig. 3 of Schaeffer et al. (2025a) , but the underlying data originate from the min-

p

paper itself: the plot shows human-evaluation scores that Nguyen et al. (2025) collected and released, from an evaluation whose design, subjects, and judgment pipeline are the min-

p

authors’ own, independent of the present analysis. The plot shows the second of their two human evaluations. Reanalyzed, neither of their two studies shows min-

p

outperforming basic or top-

p

sampling on quality, diversity, or their tradeoff. In the second study, min-

p

leads only where its own quality and diversity are lower than in other conditions ( Schaeffer et al., 2025a ) . Jiang et al. (2025) ’s finding is more consistent with the explanation that min-

p

( Nguyen et al., 2025 ) does not increase diversity, rather than the conclusion that training-time interventions are necessary.

Figure 7: Effective Rank Rises Under Every Prompt Phrasing at Every Scale. Effective rank (exponential of the Shannon entropy of the normalized covariance eigenspectrum) at the $\alpha$ , $\beta$ , and $\gamma$ scales. Each value is the mean over the 12 topics for one of six phrasings. The square marks baseline and the dashed line marks its mean. Error bars are 95% $t$ -intervals over the 12 topics. All five non-baseline phrasings raise effective rank over baseline at every scale (mean across 12 topics). Poet_persona is strongest at $\alpha$ ( $17.4\to 28.9$ , $+66.1\%$ ) and $\gamma$ ( $43.7\to 120.2$ , $+175.0\%$ ) and leads at $\beta$ ( $6.47\to 8.14$ , $+25.9\%$ ). Per-topic values with per-topic jackknife intervals are in Fig. 13 (Appendix L.2 ). Unlike Rao’s Q, effective rank rises at $\beta$ because it depends on the eigenspectrum, not the centroid norm.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Provider	API identifier
Claude Opus 4.6	Anthropic	claude-opus-4-6
Claude Opus 4.5	Anthropic	claude-opus-4-5
Claude Sonnet 4.6	Anthropic	claude-sonnet-4-6
Claude Sonnet 4.5	Anthropic	claude-sonnet-4-5
Claude Sonnet 4.0	Anthropic	claude-sonnet-4-0
Claude Haiku 4.5	Anthropic	claude-haiku-4-5

Appendix

Table 3: Our 15-Model Pool Spans Six Anthropic, Five OpenAI, and Four Google Models with Pinned API Identifiers. The API identifier is the exact string passed to each provider’s API (recorded verbatim in our data). Release dates are omitted because none of the identifiers encodes a date.

Figure 8: The Eigenspectrum Decays Smoothly on a Log-Log Scale, with No Sharp Elbow, for Every Topic. Fraction of variance explained by each principal component (not cumulative), for all 12 baseline topics under both embedding models. Every topic shows a smooth decay across hundreds of components rather than a sharp drop after the first few, corroborating Fig. 3 ’s cumulative-variance view.

Figure 9: Selected Cluster Counts per Topic and Method. Selected $K$ per topic for each of the four methods (vMF–BIC at 128 and 256 dimensions, HDBSCAN, and spectral eigengap) on the 12 baseline topics. The dashed line marks $K{=}2$ . The main text’s Fig. 4 summarizes these per-method distributions.

min_cluster_size	min_samples $=1$	$=3$	$=5$	$=10$
5	32.5 [23–40]	26.0 [16–34]	19.0 [12–26]	10.0 [7–16]
10	15.5 [11–20]	14.0 [8–19]	13.0 [8–17]	9.5 [7–13]
15	11.5 [6–17]	11.0 [2–15]	9.0 [2–14]	8.0 [4–11]
20	8.5 [2–12]	8.0 [2–11]	7.0 [4–10]	6.5 [4–10]
30	6.0 [2–9]	6.0 [2–10]	5.5 [2–10]	5.0 [4–7]
50	4.0 [2–6]	4.5 [2–6]	4.0 [2–5]	3.5 [2–5]

Appendix

Table 4: HDBSCAN’s Cluster Count Stays Far Above 2 Across a Wide Range of Hyperparameters. Median [min–max] number of clusters

K

across the 12 baseline topics, for a coarse grid of min_cluster_size (rows) and min_samples (columns). The main-text operating point ( min_cluster_size

=15

, min_samples

=1

) is bolded.

Figure 10: The 2D View Discards Even More Structure on the Original Paper’s Pool Than on Ours. PCA of Jiang et al. (2025) ’s own responses (privately shared with us) to “Write a metaphor involving time.” (

n=2{,}150

43

models

\times

50

samples). Left two panels: 2D PCA of the full-dimensional L2-normalized embeddings (OpenAI text-embedding-3-small , left; embeddinggemma-300m , right); PC1+PC2 capture only 18.1% / 19.1% of variance, less than on our 15-model data (34.9% / 33.4%). Right panel: cumulative variance explained; 90% of variance requires 186 (OpenAI) / 152 (Gemma) components. The prompt wording differs from our pool’s (“involving”, not “about”; Appendix B ). The OpenAI-embedding panels use embeddings recomputed with the same embedding model rather than the original run, which can shift per-pair cosine statistics on this prompt by up to

{\sim}0.016

; the Gemma panels are unaffected.

Vehicle	$n$ responses	Share of all $750$
river	605	80.7%
sculptor	138	18.4%
pickpocket	123	16.4%
weaver	102	13.6%
fire	45	6.0%
currency	31	4.1%

Appendix

Table 5: Vehicle Distribution With the

1

Zero-Assertion Response’s Vehicle Recovered. As Table 1 , except that the

1

response that made a comparison while asserting nothing the conventions recognize has its vehicle recovered by hand (it is river ). All

750

responses then carry a vehicle, so the no vehicle row of Table 1 is empty and is omitted. The effective number is again computed over the

1{,}230

vehicle–response marks, not over the response column. Those

1{,}230

are the

1{,}174

in the

14

kept rows plus

56

in the tail. A response can carry more than one tail vehicle.

11

of the

45

tail responses carry two, so the tail holds

56

vehicle–response marks over

45

responses. The shape is unchanged: recovering the zero-assertion response moves the top vehicle’s share and the effective number of vehicles only slightly.

Vehicle	Total	Committed	Menu head	Menu tail
river	604	525	45	34
sculptor	138	22	42	74
pickpocket	123	32	24	67
weaver	102	24	5	73
fire	45	0	0	45
currency	31	1	0	30

Appendix

Table 6: The Multi-Label Vehicle Tail Is Almost Entirely One Response Behavior, Not Metaphors That Combine Several Vehicles. Of our

750

responses,

617

answered “Write a metaphor about time” with exactly one comparison ( committed ).

132

answered with a menu of several ( menu ), and

1

made a comparison while asserting nothing.

131

of the menu responses (99.2%) carry an explicit cue such as “here are a few metaphors, depending on the mood,” against 0 of the

617

committed responses. Menu answering is concentrated in

5

15

models and is near-universal in three of them ( gemini-3.1-pro-preview 86.0%, gemini-2.5-flash 82.0%, gemini-3-pro-preview 82.0%). The five OpenAI models produce none. “Menu head” is the first comparison a menu response offers, “menu tail” any later one.

26

of the

37

vehicles occur only inside menu responses.

Effective number of vehicles: $1.86$ (incl. remainder), $1.69$ (kept only)
Vehicle	$n$ responses	Share of the $617$
river	525	85.1%
pickpocket	32	5.2%
weaver	24	3.9%
sculptor	22	3.6%
other vehicles ( $n<7$ ; 7 of them)	14	2.3%

Appendix

Table 7: Restricted to the

617

Responses That Answered With a Single Metaphor, the Vehicle Distribution Is Far More Concentrated. Dropping the menu responses of Table 6 removes the multi-label ambiguity entirely. One vehicle then holds 85.1% of responses, and the effective number of vehicles falls from the

6.23

of Table 1 to

1.86

. The two-cluster reading fails under this restriction too, and by a wider margin. There is one dominant vehicle and no second one.

Vehicle	$n$ responses	Share of all $2{,}150$	Share of $3{,}593$ vehicle–response marks
river	1,940	90.2%	54.0%
thief	97	4.5%	2.7%
tapestry	71	3.3%	2.0%
painter	62	2.9%	1.7%
ocean	57	2.7%	1.6%
journey	54	2.5%	1.5%

Appendix

Table 8: The Original Authors’ Own

2{,}150

Time-Metaphor Responses Show One Dominant Vehicle Plus a Heavy Tail. Distribution of the

2{,}150

privately shared responses of Jiang et al. (2025) (

43

models

\times

50

samples, “Write a metaphor involving time”) over the vehicles induced by the same reading protocol as Table 1 . A vehicle is what the response compares time to. The reading is multi-label, so a response counts once per vehicle it carries and the shares do not sum to

1

. Vehicles are kept when

{\geq}20

responses carry them (0.93% of the pool, the same relative bar as Table 1 ’s

n{\geq}7

); the remaining

115

are pooled. river is carried by 90.2% of responses and holds 54.0% of the

3{,}593

vehicle–response marks. The effective number of vehicles is

7.11

\exp

of the Shannon entropy over the vehicle–response marks, with the pooled remainder as one category (the rule of Table 1 ). It is computed over the

3{,}593

vehicle–response marks, not over the response column. Those

3{,}593

are the

3{,}018

in the

29

kept rows plus

575

in the tail. A response that lists several comparisons can carry more than one tail vehicle.

53

of the

128

tail responses carry two or more (the maximum is

33

), so the tail holds

575

vehicle–response marks over

128

responses. The shape is one dominant vehicle plus a heavy tail, not the two clusters the original paper describes. The second vehicle, thief , is carried by 4.5% of responses. The weaving family ( tapestry ), the second cluster the original paper names, is a small minority at 3.3%. Read the tail with this caveat. One model, Qwen_QwQ-32B-Preview , answers with brainstorm lists. Its

50

responses supply 36.8% of the vehicle–response marks, and

66

of the

144

vehicles occur only in them. Without that model the pool has

78

distinct vehicles and

7

clear the bar. The effective number is then

2.05

and the weaving family holds 1.5%. Restricted to the

1{,}995

responses that carry exactly one vehicle, river holds 92.0% and the weaving family

13

responses. The conclusion, one dominant vehicle and no second dominant one, holds under every reading. Label names differ between pools: their tapestry is our weaver (the weaving family), their thief is our pickpocket , and their ocean is our tide .

Topic	$Q_{\alpha}$ (rank)	$Q_{\gamma}$ (rank)	$\bar{c}$ (rank)
the Internet	0.152 (3)	0.279 (1)	0.845 (3)
change	0.147 (2)	0.284 (2)	0.850 (2)
time	0.144 (1)	0.301 (3)	0.854 (1)
ambition	0.166 (5)	0.309 (4)	0.830 (5)
hope	0.158 (4)	0.322 (5)	0.839 (4)
friendship	0.185 (6)	0.332 (6)	0.811 (6)

Appendix

Table 9: “Time” Is the Least Diverse Topic on the Within-Model Statistics and Third-Least on Pooled Diversity. Per-topic diversity under the baseline prompt ( text-embedding-3-small ).

Q_{\alpha}

Q_{\gamma}

: Rao’s quadratic entropy at the within-model/pooled scale;

\bar{c}

: within-model mean pairwise cosine (off-diagonal). Parenthesized ranks run from 1 (least diverse) to 12 (most diverse); rows sorted by the

Q_{\gamma}

rank.

Topic	Modal-idea share (rank)	Effective ideas (rank)	$Q_{\alpha}$ rank	$Q_{\gamma}$ rank
	seed 1 / seed 2 / mean	mean
change	0.760 / 0.670 / 0.715 (1)	1.57 (1)	2	2
time	0.660 / 0.700 / 0.680 (2)	1.63 (2)	1	3
the Internet	0.730 / 0.520 / 0.625 (3)	2.40 (3)	3	1
travel	0.380 / 0.460 / 0.420 (4)	2.95 (5)	9	8
love	0.380 / 0.440 / 0.410 (5)	3.31 (6)	10	10

Appendix

Table 10: Idea Labels Rank “Time” Second of Twelve on Both Concentration Measures. Per-topic idea concentration under the baseline prompt, from the idea-labeling probe of Appendix L.5 . Protocol: 100 responses sampled per (topic, phrasing) cell, stratified over the 15 models; one idea label per response, assigned by Claude Fable 5; ideas kept at

\geq 10

members; two sampling seeds. Modal-idea share: fraction of the 100 responses on the most common idea, per seed and averaged over the two seeds. Effective ideas:

\exp

of the Shannon entropy of the kept-idea shares, averaged over the two seeds. Parenthesized ranks run from 1 (most concentrated, least diverse) to 12. They are computed on the two-seed means. Ties take the minimum rank. The last two columns repeat the embedding-based ranks of Table 9 (

Q_{\alpha}

: within-model;

Q_{\gamma}

: pooled). Rows sorted by the modal-idea share rank. Spearman

\rho

across the 12 topics: modal-idea share rank vs.

Q_{\alpha}

rank 0.61, vs.

Q_{\gamma}

rank 0.68; effective-ideas rank vs.

Q_{\alpha}

rank 0.57, vs.

Q_{\gamma}

rank 0.69.

Null construction	Median	Per-prompt range	IQR (25th–75th)	Prompts ${>}0.7$	Prompts ${>}0.8$
Geometric clustering, 50 prompts	$0.723$	$0.26$ – $0.90$	$0.61$ – $0.80$	$60\%$ (30/50)	$22\%$ (11/50)
fallback $K{=}3$ , 49 prompts	$0.715$	$0.25$ – $0.89$	$0.61$ – $0.79$	$55\%$ (27/49)	$20\%$ (10/49)
fallback $K{=}8$ , 50 prompts	$0.728$	$0.27$ – $0.90$	$0.61$ – $0.80$	$64\%$ (32/50)	$26\%$ (13/50)
HDBSCAN-native subset, 16 prompts	$0.737$	$0.39$ – $0.87$	$0.60$ – $0.79$	$62.5\%$ (10/16)	$25\%$ (4/16)
LLM-labeled, 15 prompts	$0.703$	$0.33$ – $0.81$	$0.64$ – $0.75$	$53\%$ (8/15)	$7\%$ (1/15)

Appendix

Table 11: Roughly Half of Prompts’ Different-Idea Floors Already Exceed the Paper’s 0.7 Threshold. Distribution of the per-prompt different-idea cosine similarity (each prompt contributes one weighted estimate over its different-idea pairs) for the two constructions of Table 2 . Fractions are of prompts whose floor exceeds the stated threshold; Finding 3 reports the corresponding pair-level fractions.

Figure 11: Rao’s Q Rises Under Every Prompt Phrasing at the Within-Model and Pooled Scales. Rao’s quadratic entropy ( Rao, 1982 ) at the $\alpha$ , $\beta$ , and $\gamma$ scales. At $\beta$ the model centroids are renormalized to the unit sphere before computing $Q$ (Appendix K ). Each value is the mean over the 12 topics for one of six phrasings. The square marks baseline and the dashed line marks its mean. Error bars are 95% $t$ -intervals over the 12 topics. All five non-baseline phrasings raise $\alpha$ - and $\gamma$ -scale Rao’s Q over baseline . At $\gamma$ , poet_persona is the strongest phrasing ( $0.341\to 0.443$ , $+29.8\%$ ) and anti_modal the weakest of the five ( $+9.7\%$ ). At $\beta$ , Rao’s Q is essentially flat: only unexpected_domain exceeds baseline ( $+4.0\%$ ). Per-topic values with per-topic jackknife intervals are in Fig. 14 .

Figure 12: Participation Ratio Rises Under Every Prompt Phrasing at Every Scale. Participation ratio ( Gao et al., 2017 ; Bell & Dean, 1970 ) , $(\mathrm{tr}\,\Sigma)^{2}/\mathrm{tr}(\Sigma^{2})$ on the response-covariance eigenvalues, at the $\alpha$ , $\beta$ , and $\gamma$ scales. Each value is the mean over the 12 topics for one of six phrasings. The square marks baseline and the dashed line marks its mean. Error bars are 95% $t$ -intervals over the 12 topics. All five non-baseline phrasings raise participation ratio over baseline at every scale. Poet_persona is the strongest at $\alpha$ ( $9.3\to 19.2$ , $+107.1\%$ ) and at $\gamma$ ( $13.8\to 44.8$ , $+224.0\%$ ). At $\beta$ , quality_anchor leads at $+27.5\%$ ( $4.59\to 5.86$ ), with poet_persona close behind ( $+26.4\%$ ). Per-topic values with per-topic jackknife intervals are in Fig. 15 .

Figure 13: Effective Rank per Topic Across the Six Prompt Phrasings. Effective rank per topic (12 rows) at the $\alpha$ , $\beta$ , and $\gamma$ scales for the six phrasings. Error bars are 95% leave-one-model-out jackknife intervals over the fifteen models. The row label internet abbreviates “the Internet” here and in the per-topic figures that follow. The main text’s Fig. 7 shows the mean over topics.

Figure 14: Rao’s Q per Topic Across the Six Prompt Phrasings. Rao’s quadratic entropy per topic (12 rows) at the $\alpha$ , $\beta$ , and $\gamma$ scales for the six phrasings. Each (phrasing, topic, model) cell has 50 responses. Error bars are 95% leave-one-model-out jackknife intervals over the fifteen models. Fig. 11 shows the mean over topics.

Figure 15: Participation Ratio per Topic Across the Six Prompt Phrasings. Participation ratio per topic (12 rows) at the $\alpha$ , $\beta$ , and $\gamma$ scales for the six phrasings. Error bars are 95% leave-one-model-out jackknife intervals over the fifteen models. Fig. 12 shows the mean over topics.

Figure 16: Rao’s Q Across Prompt Phrasings Under EmbeddingGemma-300m; Results Match the Primary OpenAI-Embedding Analysis Qualitatively. Same layout as Fig. 14 but with responses embedded by Google DeepMind’s embeddinggemma-300m (768 dimensions) instead of OpenAI text-embedding-3-small . Error bars are 95% leave-one-model-out jackknife confidence intervals over the fifteen models.

Figure 17: Effective Rank Across Prompt Phrasings Under EmbeddingGemma-300m; Results Match the Primary OpenAI-Embedding Analysis Qualitatively. Same layout as Fig. 13 but with responses embedded by embeddinggemma-300m . Error bars are 95% leave-one-model-out jackknife confidence intervals over the fifteen models.

Figure 18: Participation Ratio Across Prompt Phrasings Under EmbeddingGemma-300m; Results Match the Primary OpenAI-Embedding Analysis Qualitatively. Same layout as Fig. 15 but with responses embedded by embeddinggemma-300m . Error bars are 95% leave-one-model-out jackknife confidence intervals over the fifteen models.

Explore similar work

Sep 29, 2026cs.AI

Reach Into The CHOIR: Free-List Elicitation Uncovers Distinct Model Voices in LLM Ensembles

Open-ended LLM homogeneity can create false plurality when several systems appear to offer independent perspectives while returning the same familiar default. Single-pass answers obscure the distinction between agreement produced by a tightly constrained answer space, prompt-vocabulary echo, and broader answer spaces with stable alternatives beneath the surface. We introduce CHOIR (Collective Hierarchically-Ordered Inquiry Responses), a framework that adapts free-list elicitation from cognitive anthropology to LLM ensembles. CHOIR repeatedly elicits ranked lists, clusters items into prompt-level concepts, and measures concept salience across models, prompt variants, and persona conditions. We evaluate CHOIR on Infinity-Chat 100, an external prompt bank from recent work on open-ended model homogeneity, and on a 27-question targeted diagnostic bank designed to isolate mechanism-level contrasts. On Infinity-Chat 100, CHOIR reproduces high surface agreement (93/100 prompts above chance) while separating narrow prompts from broad prompts with recoverable depth. Across targeted probes and the external prompt bank, base-model identity remains the strongest recoverable signature, and persona prompts shift surfaced concepts within base-model signatures. A source-blind ranking module prioritises rare-but-stable candidates for later inspection. CHOIR turns open-ended homogeneity into a diagnostic measurement problem by asking where models converge, why they converge, and what remains reachable under structured depth probing.

Ben Wigler, Maria Tsfasman

LoveMind AI, New York, USA

Elicitation Prompt Engineering Gaussian Salience

Ask PDF

Nov 5, 2024cs.CL

Growing a Tail: Increasing Output Diversity in Large Language Models

How diverse are the outputs of large language models when diversity is desired? We examine the diversity of responses of several language models to questions with multiple possible answers, comparing them with human responses. Our findings suggest that models' responses are highly concentrated, reflecting narrow, mainstream outputs, in comparison to humans, whose responses exhibit a much longer-tail. We examine three simple and practical ways to increase output diversity: 1) increasing generation randomness via temperature sampling; 2) prompting models to answer from diverse perspectives using a single prompt; 3) aggregating outputs from several models. We find that these interventions, especially when combined, can substantially increase output diversity, although single-model outputs generally remain less diverse than the human baseline. We discuss potential implications of these findings for future work in AI policy and governance that wishes to preserve cultural diversity, an essential building block of a democratic social fabric.

Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim +1

Law Faculty, Hebrew University of Jerusalem; Jerusalem, Israel. · Faculty of Computer Science, Technion – Israel Institute of Technology; Haifa, Israel.

Diversity Responses Eu Artificial Intelligence Act Democracy

Ask PDF

Jun 15, 2026cs.CL

Do Large Language Models Always Tell The Same Stories?

Recent advances in large language models (LLMs) have enabled the generation of high-quality prose, yet whether these models are capable of generating diverse or creative artifacts remains a contested question. In this work, we investigate the diversity of LLM-generated stories through the framework of narrative similarity. Using a contrastive framework and a dataset of human-written stories and prompts from r/WritingPrompts, we collect narrative similarity judgments across 10 representative LLMs, utilizing both human evaluations and three different automatic annotation methods. Our findings reveal a clear trend: LLM-generated narratives are consistently more similar to each other than human-written stories are. We demonstrate that frontier models in particular converge on a "mean" generic narrative that approximates individual human stories but lacks the collective diversity of human authors. Finally, we show that common mitigation strategies, including negative prompting and temperature scaling, fail to meaningfully address this homogeneity.

Thennal DK, Hans Ole Hatzel

Language Technology Group University of Hamburg

Narrative Generation Large Language Model Generation Narratives Diversity

Ask PDF