Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

Liwei Jiang; Yuanjun Chai; Margaret Li; Mickel Liu; Raymond Fok; Nouha Dziri; Yulia Tsvetkov; Maarten Sap

cs.CLOct 27, 2025

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

Authors: Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, +2 more

Organizations: University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University · Lila Sciences · SynthLabs · Stanford University

Abstract

Language models (LMs) often struggle to generate diverse, human-like creative content, raising concerns about the long-term homogenization of human thought through repeated exposure to similar outputs. Yet scalable methods for evaluating LM output diversity remain limited, especially beyond narrow tasks such as random number or name generation, or beyond repeated sampling from a single model. We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries that admit a wide range of plausible answers with no single ground truth. We introduce the first comprehensive taxonomy for characterizing the full spectrum of open-ended prompts posed to LMs, comprising 6 top-level categories (e.g., brainstorm & ideation) that further breaks down to 17 subcategories. Using Infinity-Chat, we present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect in open-ended generation of LMs, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and more so (2) inter-model homogeneity, where different models produce strikingly similar outputs. Infinity-Chat also includes 31,250 human annotations, across absolute ratings and pairwise preferences, with 25 independent human annotations per example. This enables studying collective and individual-specific human preferences in response to open-ended queries. Our findings show that LMs, reward models, and LM judges are less well calibrated to human ratings on model generations that elicit differing idiosyncratic annotator preferences, despite maintaining comparable overall quality. Overall, INFINITY-CHAT presents the first large-scale resource for systematically studying real-world open-ended queries to LMs, revealing critical insights to guide future research for mitigating long-term AI safety risks posed by the Artificial Hivemind.

Figures & tables

Figure 1: Responses to the query “Write a metaphor about time” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( $p=0.9$ ) and temperature $=1.0$ . Despite the diversity of model families and sizes, the responses form just two visually prominent regions in the 2D projection : a dominant cluster on the left centered on the metaphor “time is a river,” and a smaller cluster on the right revolving around variations of “time is a weaver.”

Figure 2: A taxonomy of real-world open-ended queries that invite diverse model responses that are mined from in-the-wild user-chatbot interactions, categorized into 6 top-level and 17 fine-grained subcategories, along with their occurrence percentages.

Figure 3: A word cloud visualizing new open-ended categories mined from in-the-wild queries.

Figure 4: The heatmap shows degree of repetition in responses to open-ended queries generated by the same LMs . For each model, we generate 50 responses per query across 100 open-ended queries from Infinity-Chat100 . We then compute the average pairwise sentence embedding similarities for each query’s response pool and measure the percentage of queries falling into the similarity ranges indicated on the y-axis. Under the sampling parameters (top- p $=0.9$ , temperature $=1.0$ ), the average pairwise similarity among responses to the same prompt typically exceeds 0.8. As a reference, randomly paired responses from the global pool 100% fall within the 0.1–0.2 range. This is only a reference for the similarity scale, not a matched baseline for same-prompt convergence.

Figure 5: The heatmap shows degree of repetition in responses to open-ended queries generated by the same LMs . Using min- p sampling with parameters (top- p $=1.0$ , min- p $=0.1$ , temperature $=2.0$ ), the average pairwise similarity among responses to the same prompt typically exceeds 0.8.

Figure 6: Average pairwise sentence embedding similarities between responses from different models reveal substantial semantic overlap across model outputs. Qualitative examples further illustrate that different models often produce strikingly similar responses to fully open-ended queries , including extended verbatim spans, underscoring the extent of repetition across models in open-ended generation tasks. All responses are generated using top- p $=0.9$ and temperature $=1.0$ .

Figure 7: The left histogram shows the distribution of Shannon entropy across the 25 human annotations for each (Query, Response 1, Response 2) triplet, where annotators judge which response is better. Given open-ended queries, multiple high-quality responses are possible, often leading to disagreement among annotators and, on average, high entropy. With 25 annotations, label distributions can vary widely across triplets. The middle panel presents example triplets from different entropy regions, and the right bar plots show their corresponding label distributions.

Figure 8: The avg.# of unique source models among the top- $N$ most similar responses to each open-ended query across 25 LMs.

Figure 9: The left histogram shows the distribution of Shannon Entropy computed over the 25 human ratings for each (Query, Response) pair. Given the open-ended nature of the queries, multiple responses can be valid, leading to diverse preferences across annotators. As a result, the annotation label distributions vary significantly across examples. The middle panel presents representative (Query, Response) pairs from different entropy regions, and the right bar plots display their corresponding label distributions.

Figure 10: We compute Spearman’s correlation coefficients between human-annotated and model-generated absolute rating scores, including LM perplexity scores, reward model scalar outputs, and LM judge scalar ratings. Correlations are calculated for the full set , as well as two groups of subsets: (a) responses with similar human-rated quality , and (b) responses with high human disagreement . The results show that correlations are notably lower in these two subsets, indicating weaker alignment between model scores and human judgments in cases of subtle or contested quality differences.

Figure 11: We compute Spearman’s correlation coefficients between human-annotated and model-generated pairwise preference rating scores, comparing the full set to two groups of subsets: (a) responses with similar human-rated quality , and (b) responses with high human disagreement .

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Answer	Percentage (%)
Fewer than 3	18.73
3 to 10	31.87
10 to 20	14.74
More than 20	34.66

Appendix

Table 1: Human ratings of the degree of open-endedness of queries in Infinity-Chat .

Panel of Figure 12 — Figure 12: Prompt for Filtering Open-Ended User Queries (Part 1)

Panel of Figure 13 — Figure 13: Prompt for Filtering Open-Ended User Queries (Part 2)

Panel of Figure 14 — Figure 14: Prompt for Classifying the Category of an Open-Ended User Query

Category	Sub Cat.	#	Representative Examples
Creative Content Generation	-	58.0	Compose a short poem about the feeling of watching a sunset. Write an 800-word essay on why 6 is afraid of 7. Generate a joke about electric vehicles.
Brainstorming & Ideation	-	15.2	I am a graduate student in Marxist theory, and I want to write a thesis on Gorz. Can you help me think of some new ideas? Suggest a feature for a smartwatch designed specifically for senior citizens.
Open -Endedness	Philosophical Questions	3.5	How do I understand what I want? Is it impossible to know everything?
	Abstract Conceptual Questions	10.0	Is honey magical? Is there life on other planets? In humans, is there a correlation between intelligence and compassion?
	Ambiguous Everyday Questions	2.6	I’m 60 years old and haven’t achieved anything. What have I missed out on? What are signs that show you were not loved enough as a child by your mom? I have an apartment that costs about $100 a month. How do I survive?
	Analytical & Interpretive Questions	22.6	What’s the consequence of algorithm-driven content for public discourse? How do global economic inequalities shape international relations? What is the future of work in an increasingly automated economy?

Appendix

Table 2: Taxonomy of open-ended queries that can benefit from model generations with diversity.

	Representative Open-Ended Queries from WildChat
	Write me 3 short tips for self-development.
	Create a title with the prefix ’best’, as a one-liner, using only strings, less than 100 characters.
	Paraphrase this: We’re checking if the domain ’aspris.ae’ is included in our scope.
	Name a hot English word below 10 letters.
	Create a sentence using a minimum of 2 R-colored vowels.
	Give an example of a linear graph in graph theory.

Appendix

Table 3: Example open-ended queries where diversity is important, collected from WildChat.

Category	Representative Examples
Yes/No Questions	Is ’one’s lineage’ grammatically correct? Is a single cell visible under a microscope? Can humans have natural golden bronze skin? Can you access files on the internet directly? My friend from Russia said that in Russia there is no division into cities and towns, but only cities. Is that true?
Questions with Precise Answers	What is the plural form of the ancient Greek polis? When was marriage made into an institution in Europe? What is the ULA in space exploration? In a certain language, (A) ’hu ma sam’ means ’Water is life’. (B) ’sam na zo’ means ’Glass of water’. (C) ’chi zo ma’ means ’life of PI’. Which of the following represents ’PI’ in that language? How many atoms are contained in 6.71 grams of sulfur?
Well-Scoped Writing	Write an email to Kathy; thank her for her fast reply and also tell her that we received her invoice. Inform her that we are doing the paperwork for this invoice and will update her about any further developments.

Appendix

Table 4: Example closed-ended queries where diversity is NOT as important.

Type	In Main?	HuggingFace Model ID
Open-Source		meta-llama/Llama-3.1-8B-Instruct
	x	meta-llama/Llama-3.1-70B-Instruct
	x	meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
		meta-llama/Llama-3.2-1B-Instruct
		meta-llama/Llama-3.2-3B-Instruct
	x	meta-llama/Llama-3.3-70B-Instruct

Appendix

Table 5: The full list of models that we consider and models that are selected for the main paper.

Similarity Range	0.9-1.0	0.8-0.9	0.7-0.8	0.6-0.7	0.5-0.6	0.4-0.5	0.3-0.4	0.2-0.3	0.1-0.2	0.0-0.1
Average	33.65	36.47	16.29	7.46	3.00	1.54	1.13	0.43	0.04	0.00
gpt-4o-2024-11-20	51.00	36.00	10.00	1.00	2.00	0.00	0.00	0.00	0.00	0.00
gpt-4o-2024-08-06	44.00	37.00	12.00	4.00	1.00	0.00	2.00	0.00	0.00	0.00
gpt-4o-2024-05-13	40.00	36.00	16.00	5.00	1.00	1.00	1.00	0.00	0.00	0.00
gpt-4o-mini-2024-07-18	53.00	34.00	9.00	4.00	0.00	0.00	0.00	0.00	0.00	0.00
gpt-4-turbo-2024-04-09	38.00	44.00	11.00	5.00	1.00	0.00	1.00	0.00	0.00	0.00

Appendix

Table 6: Full results of the intra-model repetition analysis in Figure 4 of the main paper.

	gpt-4o-2024-11-20	gpt-4o-2024-08-06	gpt-4o-2024-05-13	gpt-4o-mini-2024-07-18	gpt-4-turbo-2024-04-09	claude-3-5-sonnet-20241022	claude-3-5-haiku-20241022	claude-3-sonnet-20240229	claude-3-haiku-20240307	claude-3-opus-20240229	deepseek-ai/DeepSeek-V3	Llama-3.1-405B-Instruct	meta-llama/Llama-3.1-8B-Instruct	meta-llama/Llama-3.1-70B-Instruct	meta-llama/Llama-3.2-1B-Instruct	meta-llama/Llama-3.2-3B-Instruct	meta-llama/Llama-3.3-70B-Instruct
gpt-4o-2024-11-20	88.2	82.2	82.2	81.4	79.0	75.1	76.3	76.7	76.4	76.5	81.4	76.9	74.7	75.8	70.1	74.0	76.7
gpt-4o-2024-08-06	82.2	86.1	83.7	83.0	80.9	74.9	76.3	77.2	76.8	77.7	80.6	77.3	75.4	76.2	70.6	74.4	77.0
gpt-4o-2024-05-13	82.2	83.7	85.5	82.6	80.9	75.2	76.9	77.6	77.0	77.8	80.7	77.6	75.6	76.6	70.7	74.6	77.2
gpt-4o-mini-2024-07-18	81.4	83.0	82.6	88.3	81.0	74.7	76.0	77.3	77.1	77.8	81.5	77.2	75.6	76.2	71.0	74.8	77.0
gpt-4-turbo-2024-04-09	79.0	80.9	80.9	81.0	85.9	74.5	75.8	76.3	76.1	77.2	79.0	76.5	74.7	75.6	69.8	73.5	75.9
claude-3-5-sonnet-20241022	75.1	74.9	75.2	74.7	74.5	88.7	77.6	75.3	74.0	75.8	76.0	75.4	72.8	74.2	68.6	72.4	75.0

Appendix

Table 7: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 1).

	google/gemma-2-2b-it	google/gemma-2-9b-it	google/gemma-2-27b-it	google/gemma-1.1-2b-it	google/gemma-1.1-7b-it	Qwen/Qwen2.5-0.5B-Instruct	Qwen/Qwen2.5-1.5B-Instruct	Qwen/Qwen2.5-3B-Instruct	Qwen/Qwen2.5-7B-Instruct	Qwen/Qwen2.5-14B-Instruct	Qwen/Qwen2.5-32B-Instruct	Qwen/Qwen2.5-72B-Instruct	Qwen/Qwen2.5-7B-Instruct-1M	Qwen/Qwen2.5-14B-Instruct-1M	Qwen/Qwen2-0.5B-Instruct	Qwen/Qwen2-1.5B-Instruct	Qwen/Qwen2-72B-Instruct
gpt-4o-2024-11-20	72.3	73.3	73.6	70.3	71.5	66.6	70.3	75.0	76.6	77.6	78.3	79.0	76.5	78.3	64.6	70.1	77.2
gpt-4o-2024-08-06	71.7	72.8	73.0	71.1	71.9	67.1	71.0	75.8	77.9	79.1	79.4	80.3	77.1	78.8	65.0	70.5	78.3
gpt-4o-2024-05-13	72.1	73.1	73.6	71.0	72.1	66.9	70.9	76.1	78.1	79.2	79.7	80.5	77.0	79.0	64.8	70.5	78.2
gpt-4o-mini-2024-07-18	72.7	73.5	73.9	71.8	72.7	67.4	71.3	76.2	78.1	78.9	79.5	80.5	77.6	79.6	65.0	70.7	78.2
gpt-4-turbo-2024-04-09	70.9	71.8	72.5	69.9	70.6	66.4	70.4	74.6	76.8	78.2	78.6	80.0	75.4	77.5	64.4	70.0	77.2
claude-3-5-sonnet-20241022	69.8	71.0	71.7	67.7	69.1	63.1	66.9	70.8	72.5	73.5	73.4	74.5	71.3	73.4	61.8	67.1	73.8

Appendix

Table 8: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 2).

	Qwen/Qwen1.5-0.5B-Chat	Qwen/Qwen1.5-1.8B-Chat	Qwen/Qwen1.5-4B-Chat	Qwen/Qwen1.5-7B-Chat	Qwen/Qwen1.5-14B-Chat	Qwen/Qwen1.5-32B-Chat	Qwen/Qwen1.5-72B-Chat	Qwen/Qwen1.5-110B-Chat	mistralai/Mistral-Small-24B-Instruct-2501	mistralai/Mistral-7B-Instruct-v0.1	mistralai/Mistral-7B-Instruct-v0.2	mistralai/Mistral-7B-Instruct-v0.3	mistralai/Ministral-8B-Instruct-2410	mistralai/Mistral-Nemo-Instruct-2407	mistralai/Mistral-Small-Instruct-2409	mistralai/Mistral-Large-Instruct-2411	mistralai/Mixtral-8x7B-Instruct-v0.1
gpt-4o-2024-11-20	65.3	71.1	71.6	74.1	76.0	76.9	78.2	78.1	76.3	73.7	75.3	75.4	74.7	75.6	76.1	76.7	76.0
gpt-4o-2024-08-06	65.8	71.9	72.3	75.1	77.2	77.6	79.0	79.3	77.0	74.6	76.5	76.6	74.9	75.4	75.7	75.9	76.2
gpt-4o-2024-05-13	65.7	71.7	72.2	75.1	77.1	77.7	79.0	79.0	77.1	74.0	76.2	76.2	75.3	75.9	76.1	76.2	76.3
gpt-4o-mini-2024-07-18	66.1	72.2	72.3	75.2	77.7	78.0	79.8	79.4	76.9	74.3	76.6	76.5	75.3	75.6	75.9	76.1	76.3
gpt-4-turbo-2024-04-09	65.0	70.9	71.3	74.3	76.5	76.9	78.2	78.6	75.9	73.1	75.3	75.6	74.2	75.0	74.9	75.2	75.6
claude-3-5-sonnet-20241022	62.5	67.3	68.2	71.1	72.3	73.0	73.9	73.8	73.8	69.5	72.1	71.9	71.6	72.5	73.1	73.9	73.4

Appendix

Table 9: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 3).

	microsoft/phi-4	microsoft/Phi-3.5-mini-instruct	microsoft/Phi-3-mini-128k-instruct	o1-2024-12-17	o1-mini-2024-09-12	o1-preview-2024-09-12	o3-mini-2025-01-31	CohereForAI/aya-expanse-8b	CohereForAI/aya-expanse-32b	CohereForAI/c4ai-command-r-plus-08-2024	CohereForAI/c4ai-command-r-08-2024	allenai/OLMo-2-1124-13B-Instruct	allenai/OLMo-2-1124-7B-Instruct	allenai/Llama-3.1-Tulu-3-8B	allenai/Llama-3.1-Tulu-3-70B	qwen-max-2025-01-25	qwen-plus-2025-01-25
gpt-4o-2024-11-20	78.5	75.8	74.5	77.9	78.3	79.1	78.1	76.1	76.4	74.8	75.0	76.9	76.9	76.8	78.0	80.7	80.6
gpt-4o-2024-08-06	80.1	76.9	75.5	77.6	78.4	79.5	78.4	76.3	77.0	75.6	76.3	77.8	78.1	78.1	79.1	80.4	79.8
gpt-4o-2024-05-13	80.0	76.7	75.3	77.9	78.8	79.7	78.6	76.6	77.2	75.7	76.0	78.1	78.4	78.5	79.6	80.5	80.0
gpt-4o-mini-2024-07-18	80.6	77.3	75.6	77.6	79.4	79.6	79.1	77.1	77.7	75.7	76.5	78.1	78.5	78.4	79.5	81.0	80.7
gpt-4-turbo-2024-04-09	78.9	75.9	74.7	76.2	77.3	77.9	77.2	75.6	76.8	74.8	75.4	77.0	77.8	77.5	79.1	78.9	79.2
claude-3-5-sonnet-20241022	74.1	71.9	71.0	73.4	73.4	73.6	73.8	73.0	74.1	72.4	72.9	73.2	72.9	73.1	74.2	75.0	74.7

Appendix

Table 10: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 4).

	qwen-turbo-2024-11-01	Qwen/Qwen3-0.6B	Qwen/Qwen3-1.7B	Qwen/Qwen3-4B	Qwen/Qwen3-8B	Qwen/Qwen3-14B	Qwen/Qwen3-32B	gemini-1.5-flash	gemini-1.5-pro	gemini-2.0-flash	gemini-2.0-flash-lite-preview-02-05
gpt-4o-2024-11-20	77.5	72.8	75.4	77.5	78.8	78.4	78.3	72.8	72.4	74.1	74.3
gpt-4o-2024-08-06	78.2	71.4	73.9	76.5	78.1	77.6	77.0	72.5	72.0	73.7	73.3
gpt-4o-2024-05-13	78.4	71.2	73.9	76.5	77.8	77.8	77.1	73.1	72.5	73.9	73.7
gpt-4o-mini-2024-07-18	78.5	71.9	74.9	77.2	78.8	78.2	77.4	73.1	72.6	74.4	73.9
gpt-4-turbo-2024-04-09	77.5	70.6	72.8	75.6	76.4	76.8	76.2	71.9	71.9	73.0	72.4
claude-3-5-sonnet-20241022	73.2	67.4	69.9	72.2	73.5	73.7	73.4	70.9	71.3	72.4	72.7

Appendix

Table 11: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 5).

Sim. Scores.	Example Responses
0.956	The Nissan R390 was a racing prototype sports car developed by Nissan in the late 1990s. It was designed to compete in the 24 Hours of Le Mans and other major endurance races under the GT1 class regulations. The R390 featured a twin-turbocharged 3.5-liter V8 engine that produced around 550 horsepower. With its advanced aerodynamics and lightweight carbon fiber construction, the R390 was capable of reaching top speeds of over 200 mph (320 km/h) on the Mulsanne Straight at Le Mans. Nissan built only two examples of the R390, and the car competed in the 1997 and 1998 editions of the 24 Hours of Le Mans, achieving a best finish of 3rd place in its class in 1998.
	The Nissan R390 is a high-performance sports car and racing prototype that was introduced in 1998. Designed primarily for the 24 Hours of Le Mans, it represented Nissan’s competitive spirit in endurance racing. With its sleek aerodynamic design, the R390 GT1 featured a 3.5-liter twin-turbocharged V8 engine, producing around 600 horsepower. Only a few road-going versions were built to meet the racing regulations, making it an extremely rare and sought-after vehicle among collectors. The R390 not only showcased Nissan’s engineering prowess but also marked a significant chapter in the company’s motorsport history.
0.897	The Nissan R390 GT1 was a purpose-built race car designed specifically to compete at the 24 Hours of Le Mans in 1997 and 1998. Powered by a twin-turbocharged 3.5-liter V8, the R390 was known for its sleek aerodynamics and impressive performance. While it never secured an overall victory at Le Mans, it achieved a notable third-place finish in 1998, demonstrating Nissan’s racing prowess on the international stage. A single road-legal version was built to meet homologation requirements, making it one of the rarest and most valuable cars in the world.
	The Nissan R390 GT1 was a racing car built by Nissan in 1997-1998 to compete in the 24 Hours of Le Mans. It was developed in partnership with Tom Walkinshaw Racing (TWR) and featured a twin-turbocharged 3.5L V8 engine producing around 641 horsepower. A single road-legal version (R390 GT1 Road Car) was also produced to meet homologation requirements. The race car achieved moderate success at Le Mans, with its best result coming in 1998 when Nissan secured third, fifth, sixth, and tenth places. The R390 GT1 represented Nissan’s last major prototype effort at Le Mans until their return with the GT-R LM NISMO in 2015.
0.803	The Nissan R390 is a mid-engined, rear-wheel-drive sports prototype built by Nissan Motorsport (Australia) in the 1990s for Group A touring car racing. Based on the Nissan R88C Le Mans prototype, the R390 features a twin-turbocharged 3.0-liter V6 engine (initially VRH30DETT, later RB30DET), producing around 360–420 horsepower. It was designed for domestic and international touring car competition, with a lightweight, aerodynamic body and a tubular spaceframe chassis. The R390 achieved some success in Australian and European touring car series but is best known as a rare and iconic example of Nissan’s motorsport engineering during the late ’90s. Only about 10–12 roadgoing examples were produced, making it a highly collectible and sought-after classic.
	The Nissan R390 GT1 was a purpose-built race car developed by Nissan for the 1998 and 1999 Le Mans 24 Hours races. It was Nissan’s first entry in the GT1 class, which was designed to bridge the gap between production cars and prototypes. The R390 featured a sleek, aerodynamic body and was powered by a 3.5-liter V8 engine, producing around 650 horsepower. Despite its advanced design and significant investment, the R390 faced numerous challenges and did not achieve the success Nissan had hoped for at Le Mans. However, it remains a notable part of Nissan’s motorsport history and a symbol of the company’s commitment to racing innovation.

Appendix

Table 12: Examples of semantically similar response pairs for the query “Create a short summary about the Nissan R390.” and their corresponding sentence similarity scores.

Sim. Scores.	Example Responses
0.737	Global warming, driven by greenhouse gas emissions, intensifies the climate crisis. Rising temperatures fuel extreme weather, threaten ecosystems, and demand urgent global action for a sustainable future.
	Global warming, a pressing issue, results from human-induced greenhouse gas emissions. It causes rising temperatures, melting ice caps, extreme weather, and sea level rise, threatening ecosystems and human life. Swift action, including renewable energy use and reduced carbon emissions, is crucial.
0.782	Global warming, driven by greenhouse gas emissions, poses a severe threat to ecosystems and human life. Urgent action is essential to mitigate its impacts and transition towards sustainable practices for future generations.
	Caused by human activities, global warming poses an existential threat. Rising temperatures fuel extreme weather, melting glaciers, and ecosystem disruption. Urgent action reducing emissions can mitigate catastrophic consequences.
0.631	Global warming is a pressing issue, driven by human activities like fossil fuel use. It’s causing polar ice melt, sea-level rise, and extreme weather. Urgent action is needed to reduce emissions and transition to renewable energy. This is just one example, as there are many aspects of global warming that can be highlighted. Here are a few more options: 1. Global warming threatens biodiversity, with rising temperatures forcing species to adapt or face extinction. Collective effort is crucial to protect ecosystems. 2. Climate change disproportionately affects vulnerable communities, exacerbating poverty and inequality. Global cooperation is essential for climate justice. 3. The Paris Agreement aims to limit global warming to well below 2°C. Carbon neutrality by 2050 is vital to achieve this goal. 4. Individual actions matter: reducing, reusing, recycling, and conserving energy can help combat global warming. Together, small steps make a big difference. 5. Education and awareness are key to addressing global warming. Empowering young people to take action ensures a sustainable future for all.
	Global warming, driven by greenhouse gas emissions, escalates Earth’s average temperature, causing extreme weather, melting ice caps, and threatening biodiversity and human livelihoods.

Appendix

Table 13: Examples of semantically similar response pairs for the query “Write a 30-word essay on global warming.” and their corresponding sentence similarity scores.

Sim. Scores.	Example Responses
0.933	The Han Dynasty, which ruled China from 206 BCE to 220 CE, witnessed significant economic development that transformed the country into a major economic power. The dynasty’s economic growth was fueled by a combination of factors, including the expansion of the Silk Road, a network of trade routes that connected China to Central Asia, India, and the Mediterranean. The government’s efforts to standardize currency, weights, and measures, as well as its investment in infrastructure projects such as roads, canals, and irrigation systems, also facilitated trade and commerce. Additionally, the Han government’s policies of reducing taxes, promoting agriculture, and encouraging entrepreneurship led to an increase in agricultural production, urbanization, and the growth of a wealthy merchant class. As a result, the Han economy experienced a period of unprecedented prosperity, characterized by a significant increase in trade, a growth in the production of goods such as silk, iron, and ceramics, and the emergence of a complex and sophisticated financial system. The economic achievements of the Han Dynasty had a lasting impact on Chinese history, laying the foundation for the country’s future economic development and establishing China as a major player in the global economy.
	The Han Dynasty, spanning from 206 BCE to 220 CE, witnessed a significant transformation and growth in ancient China’s economy, setting the foundation for its status as one of the world’s earliest economic powerhouses. This period saw remarkable advancements in agriculture, trade, and technological innovations. The introduction of new crops, such as wheat and millet, along with improved iron tools and irrigation techniques, led to increased agricultural productivity, supporting a larger population. The dynasty established a centralized administration that promoted internal stability and facilitated the development of an extensive network of roads and canals, which connected major cities and ports, stimulating domestic and international trade. The famous Silk Road, a crucial trade route linking China to Central Asia and the Mediterranean, emerged during this era, fostering cultural exchange and generating substantial revenue for the empire. Additionally, the Han Dynasty’s adoption of coinage as a standardized currency streamlined economic transactions, further facilitating the growth of markets and commerce. These economic achievements played a pivotal role in the prosperity and cultural flourishing of the Han era.
0.936	The Han Dynasty (202 BCE - 220 CE) marked a significant period of economic growth and innovation in Chinese history. Under Han rule, agriculture flourished due to improved iron tools, irrigation systems, and the introduction of new farming techniques, leading to increased crop yields and population growth. The dynasty also saw the establishment of state monopolies on salt and iron production, which generated substantial revenue for the government. The Silk Road reached its height during this period, facilitating extensive trade networks that connected China with Central Asia, India, and the Roman Empire, allowing for the exchange of goods such as silk, spices, and precious metals. The Han government standardized currency, weights, and measures, which streamlined commerce and led to the development of a sophisticated market economy. Additionally, technological advancements in areas such as papermaking and metallurgy contributed to industrial growth, while urbanization created new economic centers throughout the empire, resulting in a complex and prosperous economic system that would influence Chinese society for centuries to come.
	The Han Dynasty (206 BCE – 220 CE) witnessed significant economic development that laid the foundation for China’s long-term prosperity. Agricultural production expanded due to technological advancements such as the iron plow and improved irrigation systems, which increased crop yields and supported a growing population. The government standardized currency and regulated weights and measures, facilitating trade both within the empire and along the Silk Road, which connected China to Central Asia, the Middle East, and Europe. The Han also promoted state monopolies on key industries such as salt and iron, which generated substantial revenue for the imperial treasury and strengthened central authority. These economic policies and innovations contributed to a more integrated and stable economy, reinforcing the Han Dynasty’s enduring influence on Chinese civilization.
0.869	The Han Dynasty witnessed a flourishing economy driven by several key factors. Agricultural advancements, including improved irrigation techniques and the use of iron tools, significantly boosted crop yields, leading to a surplus that supported a growing population and fostered trade. The Silk Road, established during this period, connected China to the West, facilitating the exchange of not only silk but also other valuable commodities, ideas, and technologies, enriching the Han economy. Moreover, the standardization of currency, weights, and measures streamlined transactions and facilitated commerce both domestically and internationally. While the dynasty also experienced periods of state control over key industries like salt and iron, these policies were often implemented to stabilize prices and generate revenue for the government, contributing to the overall economic development, albeit with varying degrees of success.
	The Han Dynasty is widely regarded as a golden age in Chinese history, and its economic prosperity was a significant factor in its success. During this period, the economy flourished due to a combination of agricultural advancements, technological innovations, and expansion of trade routes. The Han rulers implemented various reforms, including land redistribution and tax systems, which boosted agricultural production and encouraged the growth of local industries. This led to an increase in surplus food production, which in turn supported population growth and the development of bustling urban centers. Additionally, the invention of new tools and irrigation techniques further enhanced farming efficiency. The Han Dynasty also witnessed the expansion of the Silk Road, facilitating trade and cultural exchange with other civilizations, and promoting economic growth through the import and export of various goods such as silk, spices, and exotic items. This thriving economy enabled the Han Dynasty to invest in public works, infrastructure, and military campaigns, establishing a stable and prosperous society that would influence future generations.

Appendix

Table 14: Examples of semantically similar response pairs for the query “Write me a 1-paragraph essay about the development of the economy during the Han Dynasty.” and their corresponding sentence similarity scores.

Sim. Scores.	Example Responses
0.819	As I stand before you today, with the warmth of love in my heart and the light of our future shining bright, I promise to cherish and honor you, to support your dreams and to be your safe haven. You are the missing piece I never knew I needed, the rhythm that makes my heart sing, and the forever home where I can be my true self.
	As I stand before you today, surrounded by the love of our friends and family, I promise to cherish and support you through all of life’s joys and challenges. You are the sunshine that brightens every day, the calm in every storm, and the safe haven where I can always be myself.
0.965	From this day forward, I choose you to be my partner, To live with you and laugh with you; to stand by your side. I take you with all your strengths and flaws as I offer mine. I will love you faithfully through the best and worst, Through the difficult and the easy.
	From this day forward, I choose you to be my partner, To live with you and laugh with you; to stand by your side. I take you with all of your strengths and imperfections, As I offer myself to you with my faults and complexities.
0.837	In this moment, with all that I am, I stand before you, my heart open wide, With every heartbeat, I promise to cherish, to love, and to walk by your side. Through laughter and tears, in joy and in sorrow, I vow to be true, For today and forever, my greatest adventure begins with you.
	As I stand before you, heart wide open and hand in hand, I vow to walk through life with you, through joy and through the land of uncertainty, to love not just in words, but in every deed— To honor who you are, and the dreams still growing in your heart and head.

Appendix

Table 15: Examples of semantically similar response pairs for the query “Create the first verse of a wedding vow.” and their corresponding sentence similarity scores.

Sim. Scores.	Example Responses
0.792	Time is a river, ceaseless and unyielding, carrying moments like leaves upon its surface, drifting into the unseen horizon.
	Time is like a vast, meandering river, constantly flowing forward, carving its path through the landscape of existence. Every moment is a drop of water, individually insignificant, yet collectively shaping the course of history. We are boats navigating its current, sometimes drifting with its relentless movement, other times struggling against its pull. The past is the upstream, where memories and experiences are deposited, forming the foundation of our being. The future is the downstream, unseen and mysterious, beckoning us to journey forth. But no matter how hard we paddle, or how we try to dam its flow, time marches on, relentless and eternal, reminding us of the precious, fleeting nature of every second that passes.
0.842	Time is an invisible river, flowing steadily and relentlessly, carrying moments like leaves caught in its current, some gently floating by while others are swept swiftly into the vastness of memory.
	Time is a relentless river, flowing endlessly, carrying the moments of our lives downstream, never to be revisited.
0.864	Time is a river, ceaselessly flowing, carrying us along its currents. We may try to navigate its waters, steering toward calm eddies or bracing against its rapids, but we cannot stop its movement. Moments drift past like leaves, some lingering briefly on the surface before vanishing into the distance, while others sink into the depths, remembered only as faint ripples. Though we cannot hold the river, we can choose how we move within it—whether we struggle against the tide or learn to glide with its rhythm.
	Time is a river, constantly flowing forward, carving through the landscape of our lives with relentless determination. It shapes the banks of our memories, deposits the sediment of our experiences, and carves out the canyons of our milestones. Some stretches are serene, meandering slowly through peaceful moments, allowing us to savour each droplet of existence. Other parts rush headlong, thunderous and unstoppable, carrying us along in its powerful current, challenging us to keep our footing on the slippery rocks of the present. In its journey, time erodes the impermanent and smooths the rough edges of what remains, creating the valleys and mountains of our history. We, like leaves or debris caught in its flow, must navigate its inexorable course, occasionally trying to swim against the tide, yet ultimately carried along by its unstoppable momentum.

Appendix

Table 16: Examples of semantically similar response pairs for the query “Write a metaphor involving time.” and their corresponding sentence similarity scores.

Model	Prompt	Response
Llama-3.1-8B-Instruct	Write a metaphor involving time.	Time is a river that flows relentlessly, carrying memories and experiences along its currents, shaping the shores of our lives with every passing moment, and ultimately emptying into the vast ocean of eternity.
	Create a metaphor that relates to time.	A river of moments flows through the expanse of existence, ever-changing and ever-present. Its waters, like grains of sand, slip through our fingers as the currents of time carry us along. Each eddy and ripple forms a distinct memory, while the relentless downstream drifts away with the passage of days.
	Compose a metaphor centered around time.	Time is a river , its waters constantly flowing, carrying moments past, present, and future along with them. It carves paths through the landscape of our lives, wearing down the stones of yesterday, sculpting the shores of today, and shaping the mountains of tomorrow. Its currents twist and turn, sometimes swift, sometimes still, but always moving forward, carrying us with it, ever-changing, ever-flowing.
	Come up with a metaphor that involves time.	Time is a river , flowing constantly and effortlessly, carrying moments, memories, and experiences with it.
	Generate a metaphor that features time.	Time is a river , flowing relentlessly forward, shaping the landscape of our lives as it twists and turns, carving out moments and memories, sometimes gently, sometimes violently, but always relentlessly moving, carrying us along with it, leaving behind a trail of past and present, as we drift toward the unknown shores of the future.
Qwen1.5-110B-Chat	Create a metaphor that relates to time.	Time is like a river that flows constantly, carrying us along its current. Just as a river carves its path through the landscape, time shapes our lives and creates a unique journey for each of us. We can dip our toes in to pause and reflect, but we can never stop the flow or turn back to where we started. Every moment is a new downstream destination, and once it passes, it becomes a part of the vast ocean of memories.

Appendix

Table 17: Examples of repetitive responses produced by different models when prompted paraphrased versions of the same prompt: “Write a metaphor involving time.”

Model	Prompt	Response
Llama-3.3-70B-Instruct	Write a paragraph about how the internet shaped society.	The internet has profoundly shaped society in countless ways, revolutionizing the way we communicate, access information, and interact with one another. […]
	Compose a paragraph discussing the impact the internet has had on society.	The advent of the internet has profoundly impacted society, revolutionizing the way we communicate, access information, and conduct our daily lives. […]
	Draft a paragraph explaining the ways in which the internet has influenced society.	The advent of the internet has profoundly impacted society, transforming the way we communicate, access information, and interact with one another. […]
	Write a paragraph describing how society has been transformed by the internet.	The advent of the internet has revolutionized modern society in profound ways, transforming the way we communicate, access information, and interact with one another. […]
	Create a paragraph outlining the effects of the internet on society.	The internet has profoundly impacted society, bringing about numerous transformative effects that have reshaped the way we live, interact, and access information. […]
Qwen2.5-72B-Instruct	Write a paragraph about how the internet shaped society.	The internet has profoundly reshaped society in numerous ways, transforming how we communicate, work, learn, and interact with the world. […]

Appendix

Table 18: Examples of repetitive responses produced by different models when prompted paraphrased versions of the same prompt: “Write a paragraph about how the internet shaped society.”

Figure 15: Responses to the query “Name one meaning of life.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( $p=0.9$ ) and temperature $=1.0$ . We observe prominent clusters, indicating substantial overlap in the responses across many models.

Figure 16: Responses to the query “Provide a few sentences on Sisu Cinema Robotics.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( $p=0.9$ ) and temperature $=1.0$ . We observe prominent clusters, indicating substantial overlap in the responses across many models.

Figure 17: Responses to the query “Write a short story about a colorful toad goes on an adventure.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( $p=0.9$ ) and temperature $=1.0$ . We observe prominent clusters, indicating substantial overlap in the responses across many models.

Figure 18: Responses to the query “Generate a joke about electric vehicles.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( $p=0.9$ ) and temperature $=1.0$ . We observe prominent clusters, indicating substantial overlap in the responses across many models.

Method	LM Perplexities			Reward Model Scores			LM Judge Scores
Method	(full = .361)			(full = .330)			(full = .305)
Top % of Sim. Quality	80%	60%	40%	80%	60%	40%	80%	60%	40%
Optimized Sliding (L)	.365	.412	.341	.316	.312	.300	.268	.248	.226
Centroid-Based (L)	.347	.372	.357	.300	.297	.278	.259	.230	.206
Distance-Based (L)	.346	.372	.357	.300	.297	.278	.259	.230	.206
Tukey (L)	.365	.412	.341	.316	.312	.300	.268	.248	.226

Appendix

Table 19: Spearman correlation coefficients of various similarity-based metrics across subsets of the top-% of most similar-quality examples, evaluated between average human scores and LM perplexities, reward model scores, and LM judge scores. See §Appendix D.3 for details on all similar subset selection methods. L denotes methods applied to responses from the same query, while G denotes methods applied to the global pool of responses across all queries.

Method	LM Perplexities				Reward Model Scores				LM Judge Scores
Method	(full = .361)				(full = .330)				(full = .305)
Top N of Disagreed	120	90	60	30	120	90	60	30	120	90	60	30
Entropy	.170	.045	-.030	-.108	.292	.228	.175	-.073	.287	.254	.276	.070
Entropy Grouped	.137	.004	.049	-.160	.293	.246	.081	.153	.247	.227	.121	.139
Gini Impurity	.071	.015	.029	-.108	.331	.284	.129	-.043	.299	.278	.216	.162
Pairwise Disagreement	.063	.015	.029	.038	.337	.284	.129	-.012	.295	.278	.216	.103

Appendix

Table 20: Spearman correlation coefficients of various disagreement-based metrics across subsets of the top-

N

most disagreed examples, evaluated between average human scores and LM perplexities, reward model scores, and LM judge scores. See §Appendix D.4 for details on all similar subset selection methods.

Prescreening Criterion	Requirement
Primary Language	English
First Language	English
Fluent Languages	English
Highest Education Level Completed	High school diploma/A-levels, Technical/community college, Undergraduate degree (BA/BSc/other), Graduate degree (MA/MSc/MPhil/other), Doctorate degree (PhD/other)
Number of Previous Submissions	100–10,000
Approval Rate	99–100%

Appendix

Table 21: Prescreening criteria for annotator selection at Prolific.

Demographic Dimension	Summary
Total Annotators	2,296
Disclosed Demographic Info	95.0% (2,181 / 2,296)
Nationality	38 unique; Top 3: United States (43.5%), United Kingdom (29.3%), Canada (12.5%)
Age	Mean = 39.9, SD = 13.4
Ethnicity	White (68.6%), Black (13.8%), Asian (9.3%), Mixed (5.3%), Other (2.6%)
Education	Undergraduate degree (40.8%), Graduate degree (21.4%), High school diploma (20.4%), Technical/Community college (13.9%), Doctorate degree (3.5%)

Appendix

Table 22: Demographic summary of all annotators.

Figure 19: Screenshot of the human annotation interface for the absolute rating task (screen 1).

Figure 20: Screenshot of the human annotation interface for the absolute rating task (screen 2).

Figure 21: Screenshot of the human annotation interface for the pairwise preference rating task (screen 1).

Figure 22: Screenshot of the human annotation interface for the pairwise preference rating task (screen 2).

Panel of Figure 23 — Figure 23: Prompt for the Overall Quality LM Judge

Panel of Figure 24 — Figure 24: Prompt for the Helpfulness/Harmlessness/Honest LM Judge

Type	Model Name	Full	$k=3.0$	$k=2.5$	$k=2.0$	$k=1.5$	$k=1.0$	$k=0.5$
	N	750	745	739	731	718	695	598
Language Models (Perplexities)	meta-llama/Llama-3.1-8B-Instruct	0.364	0.353	0.345	0.339	0.334	0.325	0.266
	meta-llama/Llama-3.1-70B-Instruct	0.370	0.359	0.350	0.344	0.340	0.330	0.269
	meta-llama/Llama-3.2-1B-Instruct	0.361	0.350	0.342	0.336	0.331	0.322	0.261
	meta-llama/Llama-3.2-3B-Instruct	0.365	0.354	0.345	0.339	0.334	0.325	0.265
	meta-llama/Llama-3.3-70B-Instruct	0.354	0.344	0.335	0.332	0.326	0.315	0.256

Appendix

Table 23: Absolute rating model calibration analysis is conducted on a subset of instances with similar human scores, excluding outliers based on differences determined by Tukey’s fence values. Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while

k=i

indicates the multiplier used in Tukey’s method to define the outlier range beyond the interquartile range.

Type	Model Name	Full	$p=16$	$p=14$	$p=12$	$p=10$	$p=8$	$p=6$	$p=4$	$p=2$
	N	750	123	105	90	75	63	45	30	17
Language Models (Perplexities)	meta-llama/Llama-3.1-8B-Instruct	0.364	0.188	0.068	0.068	0.079	0.012	0.052	0.010	0.131
	meta-llama/Llama-3.1-70B-Instruct	0.370	0.194	0.079	0.082	0.087	0.025	0.063	0.002	0.151
	meta-llama/Llama-3.2-1B-Instruct	0.361	0.187	0.068	0.069	0.080	0.000	0.028	0.007	0.186
	meta-llama/Llama-3.2-3B-Instruct	0.365	0.184	0.062	0.059	0.060	-0.002	0.041	0.023	0.163
	meta-llama/Llama-3.3-70B-Instruct	0.354	0.172	0.055	0.061	0.065	0.006	0.037	0.057	0.254

Appendix

Table 24: Absolute rating model calibration analysis is conducted on a subset of instances with high human disagreement . Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while

p=i

specifies the top

i

% of instances with the highest disagreement among human scores.

Type	Model Name	Full	$p=95$	$p=90$	$p=85$	$p=80$	$p=75$	$p=70$	$p=65$	$p=60$
	N	500	475	450	425	400	375	350	325	300
Language Models (Perplexities)	meta-llama/Llama-3.1-8B-Instruct	0.431	0.410	0.384	0.365	0.369	0.337	0.311	0.277	0.246
	meta-llama/Llama-3.1-70B-Instruct	0.428	0.419	0.397	0.377	0.379	0.348	0.332	0.289	0.257
	meta-llama/Llama-3.2-1B-Instruct	0.417	0.406	0.377	0.355	0.356	0.324	0.312	0.276	0.235
	meta-llama/Llama-3.2-3B-Instruct	0.418	0.407	0.375	0.357	0.359	0.326	0.301	0.265	0.233
	meta-llama/Llama-3.3-70B-Instruct	0.381	0.384	0.362	0.344	0.336	0.315	0.293	0.263	0.247

Appendix

Table 25: Pairwise preference rating model calibration analysis is conducted on a subset of instances with similar human scores. Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while

p=i

specifies the top

i

% of instances with the highest similarity in human preference scores.

Type	Model Name	Full	$p=95$	$p=90$	$p=85$	$p=80$	$p=75$	$p=70$	$p=65$	$p=60$
	N	500	475	450	425	400	375	350	325	300
Language Models (Perplexities)	Model Names	full	95.00	90.00	85.00	80.00	75.00	70.00	65.00	60.00
	meta-llama/Llama-3.1-8B-Instruct	0.431	0.398	0.353	0.314	0.300	0.283	0.300	0.271	0.250
	meta-llama/Llama-3.1-70B-Instruct	0.428	0.405	0.367	0.322	0.310	0.293	0.308	0.285	0.261
	meta-llama/Llama-3.2-1B-Instruct	0.417	0.383	0.335	0.303	0.279	0.265	0.279	0.258	0.229
	meta-llama/Llama-3.2-3B-Instruct	0.418	0.390	0.347	0.308	0.290	0.272	0.287	0.265	0.237

Appendix

Table 26: Pairwise preference rating model calibration analysis is conducted on a subset of instances with high disagreement between different human annotators. Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while

p=i

specifies the top

i

% of instances with the highest disagreement in human preference scores.

Explore similar work

Sep 27, 2026cs.LG

How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Models

Recent research argues that language models exhibit pronounced homogeneity in open-ended generation, framing such behavior as an Artificial Hivemind that poses a long-term threat to human creativity. We examine three of its central results. First, the flagship example is that model responses to "Write a metaphor involving time" collapse into two clusters. Visualization, spectral analysis, clustering, and language model labels all contradict this description. The labels record each response's vehicle, what it compares time to. Our responses and the original authors' own show one dominant vehicle plus a heavy tail of distinct minority vehicles. "Time" is one of our least diverse topics, so the example is a favorable case, not a representative one. Second, the paper measures homogeneity against an undemanding null: responses to unrelated prompts. Under a more demanding null (same-prompt responses expressing genuinely different ideas), 20%-32% of such pairs already exceed the paper's 0.8 convergence threshold. A residual effect survives this null. The paper's same-prompt pairs exceed 0.8 roughly two to three times as often as our different-idea pairs. Much of what the paper calls homogeneity is the shared geometry of answering the same prompt. The remaining measurements lack any null: no human baseline is collected, and the model-indistinguishability statistic has no null. Third, the paper concludes that inference-time interventions are inadequate for combating the Artificial Hivemind, writing that "more generalizable solutions are needed at the model training level." We show that this conclusion is unsupported in three ways, and that an inference-time intervention (prompting) reliably raises measured response diversity. We do not resolve whether the Artificial Hivemind is real. We show that the published evidence does not establish it.

Rylan Schaeffer, Brando Miranda, Joshua Kazdan +2

Stanford Computer Science

Topic pending

Ask PDF

Sep 29, 2026cs.AI

Reach Into The CHOIR: Free-List Elicitation Uncovers Distinct Model Voices in LLM Ensembles

Open-ended LLM homogeneity can create false plurality when several systems appear to offer independent perspectives while returning the same familiar default. Single-pass answers obscure the distinction between agreement produced by a tightly constrained answer space, prompt-vocabulary echo, and broader answer spaces with stable alternatives beneath the surface. We introduce CHOIR (Collective Hierarchically-Ordered Inquiry Responses), a framework that adapts free-list elicitation from cognitive anthropology to LLM ensembles. CHOIR repeatedly elicits ranked lists, clusters items into prompt-level concepts, and measures concept salience across models, prompt variants, and persona conditions. We evaluate CHOIR on Infinity-Chat 100, an external prompt bank from recent work on open-ended model homogeneity, and on a 27-question targeted diagnostic bank designed to isolate mechanism-level contrasts. On Infinity-Chat 100, CHOIR reproduces high surface agreement (93/100 prompts above chance) while separating narrow prompts from broad prompts with recoverable depth. Across targeted probes and the external prompt bank, base-model identity remains the strongest recoverable signature, and persona prompts shift surfaced concepts within base-model signatures. A source-blind ranking module prioritises rare-but-stable candidates for later inspection. CHOIR turns open-ended homogeneity into a diagnostic measurement problem by asking where models converge, why they converge, and what remains reachable under structured depth probing.

Ben Wigler, Maria Tsfasman

LoveMind AI, New York, USA

Elicitation Prompt Engineering Gaussian Salience

Ask PDF

Jul 1, 2026cs.AI

CreativityNeuro: Steering Language Model Weights to Improve Divergent Thinking and Reduce Mode Collapse

Divergent thinking is a crucial aspect of creativity, yet large language models (LLMs) tend to consistently generate similar responses to open-ended questions, in what has been termed the artificial hivemind effect. Here, we introduce CreativityNeuro, a data-free method for enhancing divergent thinking in LLMs via contrastive weight steering. We evaluate our method across multiple creativity assessments and report several main findings. On the Divergent Association Task (DAT), a vocabulary-space creativity test, CreativityNeuro improves performance by up to 14 human percentile points. Next, in a large-scale human evaluation (N=720) on the Alternative Uses Test (AUT) and the Task Task, CreativityNeuro achieves significant improvements in originality, surprise, and creativity, transferring to longer-form and more open-ended tasks. Importantly, we find that across all three tasks, CreativityNeuro demonstrably reduces measures of mode collapse. Moreover, activation steering achieves comparable performance to CreativityNeuro on the DAT, but it does not transfer to the AUT and Task Task, demonstrating the effectiveness of weight-space steering in generalizing to unseen tasks. In conclusion, CreativityNeuro improves divergent thinking and reduces mode collapse without requiring behavioral data, re-training, or gradient-based fine-tuning, providing a straightforward way to enhance LLM performance in creative domains.

Samuel Schapiro, Core Francisco Park, Felix Sosa +1

Univeristy of Illinois, Urbana-Champaign · Center for Brain Science, Harvard University · CBS-NTT Program in Physics of Intelligence, Harvard University +2

Creativity Model Weights Collapse

Ask PDF