cs.CLOct 27, 2025
SaveArtificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Organizations: University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University · Lila Sciences · SynthLabs · Stanford University
Abstract
Language models (LMs) often struggle to generate diverse, human-like creative content, raising concerns about the long-term homogenization of human thought through repeated exposure to similar outputs. Yet scalable methods for evaluating LM output diversity remain limited, especially beyond narrow tasks such as random number or name generation, or beyond repeated sampling from a single model. We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries that admit a wide range of plausible answers with no single ground truth. We introduce the first comprehensive taxonomy for characterizing the full spectrum of open-ended prompts posed to LMs, comprising 6 top-level categories (e.g., brainstorm & ideation) that further breaks down to 17 subcategories. Using Infinity-Chat, we present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect in open-ended generation of LMs, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and more so (2) inter-model homogeneity, where different models produce strikingly similar outputs. Infinity-Chat also includes 31,250 human annotations, across absolute ratings and pairwise preferences, with 25 independent human annotations per example. This enables studying collective and individual-specific human preferences in response to open-ended queries. Our findings show that LMs, reward models, and LM judges are less well calibrated to human ratings on model generations that elicit differing idiosyncratic annotator preferences, despite maintaining comparable overall quality. Overall, INFINITY-CHAT presents the first large-scale resource for systematically studying real-world open-ended queries to LMs, revealing critical insights to guide future research for mitigating long-term AI safety risks posed by the Artificial Hivemind.
Figures & tables
Figure 1: Responses to the query “Write a metaphor about time” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( ) and temperature . Despite the diversity of model families and sizes, the responses form just two visually prominent regions in the 2D projection : a dominant cluster on the left centered on the metaphor “time is a river,” and a smaller cluster on the right revolving around variations of “time is a weaver.”
Figure 2: A taxonomy of real-world open-ended queries that invite diverse model responses that are mined from in-the-wild user-chatbot interactions, categorized into 6 top-level and 17 fine-grained subcategories, along with their occurrence percentages.
Figure 3: A word cloud visualizing new open-ended categories mined from in-the-wild queries.
Figure 4: The heatmap shows degree of repetition in responses to open-ended queries generated by the same LMs . For each model, we generate 50 responses per query across 100 open-ended queries from Infinity-Chat100 . We then compute the average pairwise sentence embedding similarities for each query’s response pool and measure the percentage of queries falling into the similarity ranges indicated on the y-axis. Under the sampling parameters (top- p , temperature ), the average pairwise similarity among responses to the same prompt typically exceeds 0.8. As a reference, randomly paired responses from the global pool 100% fall within the 0.1–0.2 range. This is only a reference for the similarity scale, not a matched baseline for same-prompt convergence.
Figure 5: The heatmap shows degree of repetition in responses to open-ended queries generated by the same LMs . Using min- p sampling with parameters (top- p , min- p , temperature ), the average pairwise similarity among responses to the same prompt typically exceeds 0.8.
Figure 6: Average pairwise sentence embedding similarities between responses from different models reveal substantial semantic overlap across model outputs. Qualitative examples further illustrate that different models often produce strikingly similar responses to fully open-ended queries , including extended verbatim spans, underscoring the extent of repetition across models in open-ended generation tasks. All responses are generated using top- p and temperature .
Figure 7: The left histogram shows the distribution of Shannon entropy across the 25 human annotations for each (Query, Response 1, Response 2) triplet, where annotators judge which response is better. Given open-ended queries, multiple high-quality responses are possible, often leading to disagreement among annotators and, on average, high entropy. With 25 annotations, label distributions can vary widely across triplets. The middle panel presents example triplets from different entropy regions, and the right bar plots show their corresponding label distributions.
Figure 8: The avg.# of unique source models among the top- most similar responses to each open-ended query across 25 LMs.
Figure 9: The left histogram shows the distribution of Shannon Entropy computed over the 25 human ratings for each (Query, Response) pair. Given the open-ended nature of the queries, multiple responses can be valid, leading to diverse preferences across annotators. As a result, the annotation label distributions vary significantly across examples. The middle panel presents representative (Query, Response) pairs from different entropy regions, and the right bar plots display their corresponding label distributions.
Figure 10: We compute Spearman’s correlation coefficients between human-annotated and model-generated absolute rating scores, including LM perplexity scores, reward model scalar outputs, and LM judge scalar ratings. Correlations are calculated for the full set , as well as two groups of subsets: (a) responses with similar human-rated quality , and (b) responses with high human disagreement . The results show that correlations are notably lower in these two subsets, indicating weaker alignment between model scores and human judgments in cases of subtle or contested quality differences.
Figure 11: We compute Spearman’s correlation coefficients between human-annotated and model-generated pairwise preference rating scores, comparing the full set to two groups of subsets: (a) responses with similar human-rated quality , and (b) responses with high human disagreement .
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
| Answer | Percentage (%) |
| Fewer than 3 | 18.73 |
| 3 to 10 | 31.87 |
| 10 to 20 | 14.74 |
| More than 20 | 34.66 |
Table 1: Human ratings of the degree of open-endedness of queries in Infinity-Chat .
Figure 12: Prompt for Filtering Open-Ended User Queries (Part 1)
Figure 13: Prompt for Filtering Open-Ended User Queries (Part 2)
Figure 14: Prompt for Classifying the Category of an Open-Ended User Query
| Category | Sub Cat. | # | Representative Examples |
| Creative Content Generation | - | 58.0 | Compose a short poem about the feeling of watching a sunset. Write an 800-word essay on why 6 is afraid of 7. Generate a joke about electric vehicles. |
| Brainstorming & Ideation | - | 15.2 | I am a graduate student in Marxist theory, and I want to write a thesis on Gorz. Can you help me think of some new ideas? Suggest a feature for a smartwatch designed specifically for senior citizens. |
| Open -Endedness | Philosophical Questions | 3.5 | How do I understand what I want? Is it impossible to know everything? |
| Abstract Conceptual Questions | 10.0 | Is honey magical? Is there life on other planets? In humans, is there a correlation between intelligence and compassion? | |
| Ambiguous Everyday Questions | 2.6 | I’m 60 years old and haven’t achieved anything. What have I missed out on? What are signs that show you were not loved enough as a child by your mom? I have an apartment that costs about $100 a month. How do I survive? | |
| Analytical & Interpretive Questions | 22.6 | What’s the consequence of algorithm-driven content for public discourse? How do global economic inequalities shape international relations? What is the future of work in an increasingly automated economy? |
Table 2: Taxonomy of open-ended queries that can benefit from model generations with diversity.
| Representative Open-Ended Queries from WildChat | |
| Write me 3 short tips for self-development. | |
| Create a title with the prefix ’best’, as a one-liner, using only strings, less than 100 characters. | |
| Paraphrase this: We’re checking if the domain ’aspris.ae’ is included in our scope. | |
| Name a hot English word below 10 letters. | |
| Create a sentence using a minimum of 2 R-colored vowels. | |
| Give an example of a linear graph in graph theory. |
Table 3: Example open-ended queries where diversity is important, collected from WildChat.
| Category | Representative Examples |
| Yes/No Questions | Is ’one’s lineage’ grammatically correct? Is a single cell visible under a microscope? Can humans have natural golden bronze skin? Can you access files on the internet directly? My friend from Russia said that in Russia there is no division into cities and towns, but only cities. Is that true? |
| Questions with Precise Answers | What is the plural form of the ancient Greek polis? When was marriage made into an institution in Europe? What is the ULA in space exploration? In a certain language, (A) ’hu ma sam’ means ’Water is life’. (B) ’sam na zo’ means ’Glass of water’. (C) ’chi zo ma’ means ’life of PI’. Which of the following represents ’PI’ in that language? How many atoms are contained in 6.71 grams of sulfur? |
| Well-Scoped Writing | Write an email to Kathy; thank her for her fast reply and also tell her that we received her invoice. Inform her that we are doing the paperwork for this invoice and will update her about any further developments. |
Table 4: Example closed-ended queries where diversity is NOT as important.
| Type | In Main? | HuggingFace Model ID |
| Open-Source | meta-llama/Llama-3.1-8B-Instruct | |
| x | meta-llama/Llama-3.1-70B-Instruct | |
| x | meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo | |
| meta-llama/Llama-3.2-1B-Instruct | ||
| meta-llama/Llama-3.2-3B-Instruct | ||
| x | meta-llama/Llama-3.3-70B-Instruct |
Table 5: The full list of models that we consider and models that are selected for the main paper.
| Similarity Range | 0.9-1.0 | 0.8-0.9 | 0.7-0.8 | 0.6-0.7 | 0.5-0.6 | 0.4-0.5 | 0.3-0.4 | 0.2-0.3 | 0.1-0.2 | 0.0-0.1 |
| Average | 33.65 | 36.47 | 16.29 | 7.46 | 3.00 | 1.54 | 1.13 | 0.43 | 0.04 | 0.00 |
| gpt-4o-2024-11-20 | 51.00 | 36.00 | 10.00 | 1.00 | 2.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| gpt-4o-2024-08-06 | 44.00 | 37.00 | 12.00 | 4.00 | 1.00 | 0.00 | 2.00 | 0.00 | 0.00 | 0.00 |
| gpt-4o-2024-05-13 | 40.00 | 36.00 | 16.00 | 5.00 | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 |
| gpt-4o-mini-2024-07-18 | 53.00 | 34.00 | 9.00 | 4.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| gpt-4-turbo-2024-04-09 | 38.00 | 44.00 | 11.00 | 5.00 | 1.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 |
Table 6: Full results of the intra-model repetition analysis in Figure 4 of the main paper.
| gpt-4o-2024-11-20 | gpt-4o-2024-08-06 | gpt-4o-2024-05-13 | gpt-4o-mini-2024-07-18 | gpt-4-turbo-2024-04-09 | claude-3-5-sonnet-20241022 | claude-3-5-haiku-20241022 | claude-3-sonnet-20240229 | claude-3-haiku-20240307 | claude-3-opus-20240229 | deepseek-ai/DeepSeek-V3 | Llama-3.1-405B-Instruct | meta-llama/Llama-3.1-8B-Instruct | meta-llama/Llama-3.1-70B-Instruct | meta-llama/Llama-3.2-1B-Instruct | meta-llama/Llama-3.2-3B-Instruct | meta-llama/Llama-3.3-70B-Instruct | |
| gpt-4o-2024-11-20 | 88.2 | 82.2 | 82.2 | 81.4 | 79.0 | 75.1 | 76.3 | 76.7 | 76.4 | 76.5 | 81.4 | 76.9 | 74.7 | 75.8 | 70.1 | 74.0 | 76.7 |
| gpt-4o-2024-08-06 | 82.2 | 86.1 | 83.7 | 83.0 | 80.9 | 74.9 | 76.3 | 77.2 | 76.8 | 77.7 | 80.6 | 77.3 | 75.4 | 76.2 | 70.6 | 74.4 | 77.0 |
| gpt-4o-2024-05-13 | 82.2 | 83.7 | 85.5 | 82.6 | 80.9 | 75.2 | 76.9 | 77.6 | 77.0 | 77.8 | 80.7 | 77.6 | 75.6 | 76.6 | 70.7 | 74.6 | 77.2 |
| gpt-4o-mini-2024-07-18 | 81.4 | 83.0 | 82.6 | 88.3 | 81.0 | 74.7 | 76.0 | 77.3 | 77.1 | 77.8 | 81.5 | 77.2 | 75.6 | 76.2 | 71.0 | 74.8 | 77.0 |
| gpt-4-turbo-2024-04-09 | 79.0 | 80.9 | 80.9 | 81.0 | 85.9 | 74.5 | 75.8 | 76.3 | 76.1 | 77.2 | 79.0 | 76.5 | 74.7 | 75.6 | 69.8 | 73.5 | 75.9 |
| claude-3-5-sonnet-20241022 | 75.1 | 74.9 | 75.2 | 74.7 | 74.5 | 88.7 | 77.6 | 75.3 | 74.0 | 75.8 | 76.0 | 75.4 | 72.8 | 74.2 | 68.6 | 72.4 | 75.0 |
Table 7: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 1).
| google/gemma-2-2b-it | google/gemma-2-9b-it | google/gemma-2-27b-it | google/gemma-1.1-2b-it | google/gemma-1.1-7b-it | Qwen/Qwen2.5-0.5B-Instruct | Qwen/Qwen2.5-1.5B-Instruct | Qwen/Qwen2.5-3B-Instruct | Qwen/Qwen2.5-7B-Instruct | Qwen/Qwen2.5-14B-Instruct | Qwen/Qwen2.5-32B-Instruct | Qwen/Qwen2.5-72B-Instruct | Qwen/Qwen2.5-7B-Instruct-1M | Qwen/Qwen2.5-14B-Instruct-1M | Qwen/Qwen2-0.5B-Instruct | Qwen/Qwen2-1.5B-Instruct | Qwen/Qwen2-72B-Instruct | |
| gpt-4o-2024-11-20 | 72.3 | 73.3 | 73.6 | 70.3 | 71.5 | 66.6 | 70.3 | 75.0 | 76.6 | 77.6 | 78.3 | 79.0 | 76.5 | 78.3 | 64.6 | 70.1 | 77.2 |
| gpt-4o-2024-08-06 | 71.7 | 72.8 | 73.0 | 71.1 | 71.9 | 67.1 | 71.0 | 75.8 | 77.9 | 79.1 | 79.4 | 80.3 | 77.1 | 78.8 | 65.0 | 70.5 | 78.3 |
| gpt-4o-2024-05-13 | 72.1 | 73.1 | 73.6 | 71.0 | 72.1 | 66.9 | 70.9 | 76.1 | 78.1 | 79.2 | 79.7 | 80.5 | 77.0 | 79.0 | 64.8 | 70.5 | 78.2 |
| gpt-4o-mini-2024-07-18 | 72.7 | 73.5 | 73.9 | 71.8 | 72.7 | 67.4 | 71.3 | 76.2 | 78.1 | 78.9 | 79.5 | 80.5 | 77.6 | 79.6 | 65.0 | 70.7 | 78.2 |
| gpt-4-turbo-2024-04-09 | 70.9 | 71.8 | 72.5 | 69.9 | 70.6 | 66.4 | 70.4 | 74.6 | 76.8 | 78.2 | 78.6 | 80.0 | 75.4 | 77.5 | 64.4 | 70.0 | 77.2 |
| claude-3-5-sonnet-20241022 | 69.8 | 71.0 | 71.7 | 67.7 | 69.1 | 63.1 | 66.9 | 70.8 | 72.5 | 73.5 | 73.4 | 74.5 | 71.3 | 73.4 | 61.8 | 67.1 | 73.8 |
Table 8: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 2).
| Qwen/Qwen1.5-0.5B-Chat | Qwen/Qwen1.5-1.8B-Chat | Qwen/Qwen1.5-4B-Chat | Qwen/Qwen1.5-7B-Chat | Qwen/Qwen1.5-14B-Chat | Qwen/Qwen1.5-32B-Chat | Qwen/Qwen1.5-72B-Chat | Qwen/Qwen1.5-110B-Chat | mistralai/Mistral-Small-24B-Instruct-2501 | mistralai/Mistral-7B-Instruct-v0.1 | mistralai/Mistral-7B-Instruct-v0.2 | mistralai/Mistral-7B-Instruct-v0.3 | mistralai/Ministral-8B-Instruct-2410 | mistralai/Mistral-Nemo-Instruct-2407 | mistralai/Mistral-Small-Instruct-2409 | mistralai/Mistral-Large-Instruct-2411 | mistralai/Mixtral-8x7B-Instruct-v0.1 | |
| gpt-4o-2024-11-20 | 65.3 | 71.1 | 71.6 | 74.1 | 76.0 | 76.9 | 78.2 | 78.1 | 76.3 | 73.7 | 75.3 | 75.4 | 74.7 | 75.6 | 76.1 | 76.7 | 76.0 |
| gpt-4o-2024-08-06 | 65.8 | 71.9 | 72.3 | 75.1 | 77.2 | 77.6 | 79.0 | 79.3 | 77.0 | 74.6 | 76.5 | 76.6 | 74.9 | 75.4 | 75.7 | 75.9 | 76.2 |
| gpt-4o-2024-05-13 | 65.7 | 71.7 | 72.2 | 75.1 | 77.1 | 77.7 | 79.0 | 79.0 | 77.1 | 74.0 | 76.2 | 76.2 | 75.3 | 75.9 | 76.1 | 76.2 | 76.3 |
| gpt-4o-mini-2024-07-18 | 66.1 | 72.2 | 72.3 | 75.2 | 77.7 | 78.0 | 79.8 | 79.4 | 76.9 | 74.3 | 76.6 | 76.5 | 75.3 | 75.6 | 75.9 | 76.1 | 76.3 |
| gpt-4-turbo-2024-04-09 | 65.0 | 70.9 | 71.3 | 74.3 | 76.5 | 76.9 | 78.2 | 78.6 | 75.9 | 73.1 | 75.3 | 75.6 | 74.2 | 75.0 | 74.9 | 75.2 | 75.6 |
| claude-3-5-sonnet-20241022 | 62.5 | 67.3 | 68.2 | 71.1 | 72.3 | 73.0 | 73.9 | 73.8 | 73.8 | 69.5 | 72.1 | 71.9 | 71.6 | 72.5 | 73.1 | 73.9 | 73.4 |
Table 9: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 3).
| microsoft/phi-4 | microsoft/Phi-3.5-mini-instruct | microsoft/Phi-3-mini-128k-instruct | o1-2024-12-17 | o1-mini-2024-09-12 | o1-preview-2024-09-12 | o3-mini-2025-01-31 | CohereForAI/aya-expanse-8b | CohereForAI/aya-expanse-32b | CohereForAI/c4ai-command-r-plus-08-2024 | CohereForAI/c4ai-command-r-08-2024 | allenai/OLMo-2-1124-13B-Instruct | allenai/OLMo-2-1124-7B-Instruct | allenai/Llama-3.1-Tulu-3-8B | allenai/Llama-3.1-Tulu-3-70B | qwen-max-2025-01-25 | qwen-plus-2025-01-25 | |
| gpt-4o-2024-11-20 | 78.5 | 75.8 | 74.5 | 77.9 | 78.3 | 79.1 | 78.1 | 76.1 | 76.4 | 74.8 | 75.0 | 76.9 | 76.9 | 76.8 | 78.0 | 80.7 | 80.6 |
| gpt-4o-2024-08-06 | 80.1 | 76.9 | 75.5 | 77.6 | 78.4 | 79.5 | 78.4 | 76.3 | 77.0 | 75.6 | 76.3 | 77.8 | 78.1 | 78.1 | 79.1 | 80.4 | 79.8 |
| gpt-4o-2024-05-13 | 80.0 | 76.7 | 75.3 | 77.9 | 78.8 | 79.7 | 78.6 | 76.6 | 77.2 | 75.7 | 76.0 | 78.1 | 78.4 | 78.5 | 79.6 | 80.5 | 80.0 |
| gpt-4o-mini-2024-07-18 | 80.6 | 77.3 | 75.6 | 77.6 | 79.4 | 79.6 | 79.1 | 77.1 | 77.7 | 75.7 | 76.5 | 78.1 | 78.5 | 78.4 | 79.5 | 81.0 | 80.7 |
| gpt-4-turbo-2024-04-09 | 78.9 | 75.9 | 74.7 | 76.2 | 77.3 | 77.9 | 77.2 | 75.6 | 76.8 | 74.8 | 75.4 | 77.0 | 77.8 | 77.5 | 79.1 | 78.9 | 79.2 |
| claude-3-5-sonnet-20241022 | 74.1 | 71.9 | 71.0 | 73.4 | 73.4 | 73.6 | 73.8 | 73.0 | 74.1 | 72.4 | 72.9 | 73.2 | 72.9 | 73.1 | 74.2 | 75.0 | 74.7 |
Table 10: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 4).
| qwen-turbo-2024-11-01 | Qwen/Qwen3-0.6B | Qwen/Qwen3-1.7B | Qwen/Qwen3-4B | Qwen/Qwen3-8B | Qwen/Qwen3-14B | Qwen/Qwen3-32B | gemini-1.5-flash | gemini-1.5-pro | gemini-2.0-flash | gemini-2.0-flash-lite-preview-02-05 | |
| gpt-4o-2024-11-20 | 77.5 | 72.8 | 75.4 | 77.5 | 78.8 | 78.4 | 78.3 | 72.8 | 72.4 | 74.1 | 74.3 |
| gpt-4o-2024-08-06 | 78.2 | 71.4 | 73.9 | 76.5 | 78.1 | 77.6 | 77.0 | 72.5 | 72.0 | 73.7 | 73.3 |
| gpt-4o-2024-05-13 | 78.4 | 71.2 | 73.9 | 76.5 | 77.8 | 77.8 | 77.1 | 73.1 | 72.5 | 73.9 | 73.7 |
| gpt-4o-mini-2024-07-18 | 78.5 | 71.9 | 74.9 | 77.2 | 78.8 | 78.2 | 77.4 | 73.1 | 72.6 | 74.4 | 73.9 |
| gpt-4-turbo-2024-04-09 | 77.5 | 70.6 | 72.8 | 75.6 | 76.4 | 76.8 | 76.2 | 71.9 | 71.9 | 73.0 | 72.4 |
| claude-3-5-sonnet-20241022 | 73.2 | 67.4 | 69.9 | 72.2 | 73.5 | 73.7 | 73.4 | 70.9 | 71.3 | 72.4 | 72.7 |
Table 11: Full results of the inter-model repetition analysis in Figure 6 of the main paper (Part 5).
| Sim. Scores. | Example Responses |
| 0.956 | The Nissan R390 was a racing prototype sports car developed by Nissan in the late 1990s. It was designed to compete in the 24 Hours of Le Mans and other major endurance races under the GT1 class regulations. The R390 featured a twin-turbocharged 3.5-liter V8 engine that produced around 550 horsepower. With its advanced aerodynamics and lightweight carbon fiber construction, the R390 was capable of reaching top speeds of over 200 mph (320 km/h) on the Mulsanne Straight at Le Mans. Nissan built only two examples of the R390, and the car competed in the 1997 and 1998 editions of the 24 Hours of Le Mans, achieving a best finish of 3rd place in its class in 1998. |
| The Nissan R390 is a high-performance sports car and racing prototype that was introduced in 1998. Designed primarily for the 24 Hours of Le Mans, it represented Nissan’s competitive spirit in endurance racing. With its sleek aerodynamic design, the R390 GT1 featured a 3.5-liter twin-turbocharged V8 engine, producing around 600 horsepower. Only a few road-going versions were built to meet the racing regulations, making it an extremely rare and sought-after vehicle among collectors. The R390 not only showcased Nissan’s engineering prowess but also marked a significant chapter in the company’s motorsport history. | |
| 0.897 | The Nissan R390 GT1 was a purpose-built race car designed specifically to compete at the 24 Hours of Le Mans in 1997 and 1998. Powered by a twin-turbocharged 3.5-liter V8, the R390 was known for its sleek aerodynamics and impressive performance. While it never secured an overall victory at Le Mans, it achieved a notable third-place finish in 1998, demonstrating Nissan’s racing prowess on the international stage. A single road-legal version was built to meet homologation requirements, making it one of the rarest and most valuable cars in the world. |
| The Nissan R390 GT1 was a racing car built by Nissan in 1997-1998 to compete in the 24 Hours of Le Mans. It was developed in partnership with Tom Walkinshaw Racing (TWR) and featured a twin-turbocharged 3.5L V8 engine producing around 641 horsepower. A single road-legal version (R390 GT1 Road Car) was also produced to meet homologation requirements. The race car achieved moderate success at Le Mans, with its best result coming in 1998 when Nissan secured third, fifth, sixth, and tenth places. The R390 GT1 represented Nissan’s last major prototype effort at Le Mans until their return with the GT-R LM NISMO in 2015. | |
| 0.803 | The Nissan R390 is a mid-engined, rear-wheel-drive sports prototype built by Nissan Motorsport (Australia) in the 1990s for Group A touring car racing. Based on the Nissan R88C Le Mans prototype, the R390 features a twin-turbocharged 3.0-liter V6 engine (initially VRH30DETT, later RB30DET), producing around 360–420 horsepower. It was designed for domestic and international touring car competition, with a lightweight, aerodynamic body and a tubular spaceframe chassis. The R390 achieved some success in Australian and European touring car series but is best known as a rare and iconic example of Nissan’s motorsport engineering during the late ’90s. Only about 10–12 roadgoing examples were produced, making it a highly collectible and sought-after classic. |
| The Nissan R390 GT1 was a purpose-built race car developed by Nissan for the 1998 and 1999 Le Mans 24 Hours races. It was Nissan’s first entry in the GT1 class, which was designed to bridge the gap between production cars and prototypes. The R390 featured a sleek, aerodynamic body and was powered by a 3.5-liter V8 engine, producing around 650 horsepower. Despite its advanced design and significant investment, the R390 faced numerous challenges and did not achieve the success Nissan had hoped for at Le Mans. However, it remains a notable part of Nissan’s motorsport history and a symbol of the company’s commitment to racing innovation. |
Table 12: Examples of semantically similar response pairs for the query “Create a short summary about the Nissan R390.” and their corresponding sentence similarity scores.
| Sim. Scores. | Example Responses |
| 0.737 | Global warming, driven by greenhouse gas emissions, intensifies the climate crisis. Rising temperatures fuel extreme weather, threaten ecosystems, and demand urgent global action for a sustainable future. |
| Global warming, a pressing issue, results from human-induced greenhouse gas emissions. It causes rising temperatures, melting ice caps, extreme weather, and sea level rise, threatening ecosystems and human life. Swift action, including renewable energy use and reduced carbon emissions, is crucial. | |
| 0.782 | Global warming, driven by greenhouse gas emissions, poses a severe threat to ecosystems and human life. Urgent action is essential to mitigate its impacts and transition towards sustainable practices for future generations. |
| Caused by human activities, global warming poses an existential threat. Rising temperatures fuel extreme weather, melting glaciers, and ecosystem disruption. Urgent action reducing emissions can mitigate catastrophic consequences. | |
| 0.631 | Global warming is a pressing issue, driven by human activities like fossil fuel use. It’s causing polar ice melt, sea-level rise, and extreme weather. Urgent action is needed to reduce emissions and transition to renewable energy. This is just one example, as there are many aspects of global warming that can be highlighted. Here are a few more options: 1. Global warming threatens biodiversity, with rising temperatures forcing species to adapt or face extinction. Collective effort is crucial to protect ecosystems. 2. Climate change disproportionately affects vulnerable communities, exacerbating poverty and inequality. Global cooperation is essential for climate justice. 3. The Paris Agreement aims to limit global warming to well below 2°C. Carbon neutrality by 2050 is vital to achieve this goal. 4. Individual actions matter: reducing, reusing, recycling, and conserving energy can help combat global warming. Together, small steps make a big difference. 5. Education and awareness are key to addressing global warming. Empowering young people to take action ensures a sustainable future for all. |
| Global warming, driven by greenhouse gas emissions, escalates Earth’s average temperature, causing extreme weather, melting ice caps, and threatening biodiversity and human livelihoods. |
Table 13: Examples of semantically similar response pairs for the query “Write a 30-word essay on global warming.” and their corresponding sentence similarity scores.
| Sim. Scores. | Example Responses |
| 0.933 | The Han Dynasty, which ruled China from 206 BCE to 220 CE, witnessed significant economic development that transformed the country into a major economic power. The dynasty’s economic growth was fueled by a combination of factors, including the expansion of the Silk Road, a network of trade routes that connected China to Central Asia, India, and the Mediterranean. The government’s efforts to standardize currency, weights, and measures, as well as its investment in infrastructure projects such as roads, canals, and irrigation systems, also facilitated trade and commerce. Additionally, the Han government’s policies of reducing taxes, promoting agriculture, and encouraging entrepreneurship led to an increase in agricultural production, urbanization, and the growth of a wealthy merchant class. As a result, the Han economy experienced a period of unprecedented prosperity, characterized by a significant increase in trade, a growth in the production of goods such as silk, iron, and ceramics, and the emergence of a complex and sophisticated financial system. The economic achievements of the Han Dynasty had a lasting impact on Chinese history, laying the foundation for the country’s future economic development and establishing China as a major player in the global economy. |
| The Han Dynasty, spanning from 206 BCE to 220 CE, witnessed a significant transformation and growth in ancient China’s economy, setting the foundation for its status as one of the world’s earliest economic powerhouses. This period saw remarkable advancements in agriculture, trade, and technological innovations. The introduction of new crops, such as wheat and millet, along with improved iron tools and irrigation techniques, led to increased agricultural productivity, supporting a larger population. The dynasty established a centralized administration that promoted internal stability and facilitated the development of an extensive network of roads and canals, which connected major cities and ports, stimulating domestic and international trade. The famous Silk Road, a crucial trade route linking China to Central Asia and the Mediterranean, emerged during this era, fostering cultural exchange and generating substantial revenue for the empire. Additionally, the Han Dynasty’s adoption of coinage as a standardized currency streamlined economic transactions, further facilitating the growth of markets and commerce. These economic achievements played a pivotal role in the prosperity and cultural flourishing of the Han era. | |
| 0.936 | The Han Dynasty (202 BCE - 220 CE) marked a significant period of economic growth and innovation in Chinese history. Under Han rule, agriculture flourished due to improved iron tools, irrigation systems, and the introduction of new farming techniques, leading to increased crop yields and population growth. The dynasty also saw the establishment of state monopolies on salt and iron production, which generated substantial revenue for the government. The Silk Road reached its height during this period, facilitating extensive trade networks that connected China with Central Asia, India, and the Roman Empire, allowing for the exchange of goods such as silk, spices, and precious metals. The Han government standardized currency, weights, and measures, which streamlined commerce and led to the development of a sophisticated market economy. Additionally, technological advancements in areas such as papermaking and metallurgy contributed to industrial growth, while urbanization created new economic centers throughout the empire, resulting in a complex and prosperous economic system that would influence Chinese society for centuries to come. |
| The Han Dynasty (206 BCE – 220 CE) witnessed significant economic development that laid the foundation for China’s long-term prosperity. Agricultural production expanded due to technological advancements such as the iron plow and improved irrigation systems, which increased crop yields and supported a growing population. The government standardized currency and regulated weights and measures, facilitating trade both within the empire and along the Silk Road, which connected China to Central Asia, the Middle East, and Europe. The Han also promoted state monopolies on key industries such as salt and iron, which generated substantial revenue for the imperial treasury and strengthened central authority. These economic policies and innovations contributed to a more integrated and stable economy, reinforcing the Han Dynasty’s enduring influence on Chinese civilization. | |
| 0.869 | The Han Dynasty witnessed a flourishing economy driven by several key factors. Agricultural advancements, including improved irrigation techniques and the use of iron tools, significantly boosted crop yields, leading to a surplus that supported a growing population and fostered trade. The Silk Road, established during this period, connected China to the West, facilitating the exchange of not only silk but also other valuable commodities, ideas, and technologies, enriching the Han economy. Moreover, the standardization of currency, weights, and measures streamlined transactions and facilitated commerce both domestically and internationally. While the dynasty also experienced periods of state control over key industries like salt and iron, these policies were often implemented to stabilize prices and generate revenue for the government, contributing to the overall economic development, albeit with varying degrees of success. |
| The Han Dynasty is widely regarded as a golden age in Chinese history, and its economic prosperity was a significant factor in its success. During this period, the economy flourished due to a combination of agricultural advancements, technological innovations, and expansion of trade routes. The Han rulers implemented various reforms, including land redistribution and tax systems, which boosted agricultural production and encouraged the growth of local industries. This led to an increase in surplus food production, which in turn supported population growth and the development of bustling urban centers. Additionally, the invention of new tools and irrigation techniques further enhanced farming efficiency. The Han Dynasty also witnessed the expansion of the Silk Road, facilitating trade and cultural exchange with other civilizations, and promoting economic growth through the import and export of various goods such as silk, spices, and exotic items. This thriving economy enabled the Han Dynasty to invest in public works, infrastructure, and military campaigns, establishing a stable and prosperous society that would influence future generations. |
Table 14: Examples of semantically similar response pairs for the query “Write me a 1-paragraph essay about the development of the economy during the Han Dynasty.” and their corresponding sentence similarity scores.
| Sim. Scores. | Example Responses |
| 0.819 | As I stand before you today, with the warmth of love in my heart and the light of our future shining bright, I promise to cherish and honor you, to support your dreams and to be your safe haven. You are the missing piece I never knew I needed, the rhythm that makes my heart sing, and the forever home where I can be my true self. |
| As I stand before you today, surrounded by the love of our friends and family, I promise to cherish and support you through all of life’s joys and challenges. You are the sunshine that brightens every day, the calm in every storm, and the safe haven where I can always be myself. | |
| 0.965 | From this day forward, I choose you to be my partner, To live with you and laugh with you; to stand by your side. I take you with all your strengths and flaws as I offer mine. I will love you faithfully through the best and worst, Through the difficult and the easy. |
| From this day forward, I choose you to be my partner, To live with you and laugh with you; to stand by your side. I take you with all of your strengths and imperfections, As I offer myself to you with my faults and complexities. | |
| 0.837 | In this moment, with all that I am, I stand before you, my heart open wide, With every heartbeat, I promise to cherish, to love, and to walk by your side. Through laughter and tears, in joy and in sorrow, I vow to be true, For today and forever, my greatest adventure begins with you. |
| As I stand before you, heart wide open and hand in hand, I vow to walk through life with you, through joy and through the land of uncertainty, to love not just in words, but in every deed— To honor who you are, and the dreams still growing in your heart and head. |
Table 15: Examples of semantically similar response pairs for the query “Create the first verse of a wedding vow.” and their corresponding sentence similarity scores.
| Sim. Scores. | Example Responses |
| 0.792 | Time is a river, ceaseless and unyielding, carrying moments like leaves upon its surface, drifting into the unseen horizon. |
| Time is like a vast, meandering river, constantly flowing forward, carving its path through the landscape of existence. Every moment is a drop of water, individually insignificant, yet collectively shaping the course of history. We are boats navigating its current, sometimes drifting with its relentless movement, other times struggling against its pull. The past is the upstream, where memories and experiences are deposited, forming the foundation of our being. The future is the downstream, unseen and mysterious, beckoning us to journey forth. But no matter how hard we paddle, or how we try to dam its flow, time marches on, relentless and eternal, reminding us of the precious, fleeting nature of every second that passes. | |
| 0.842 | Time is an invisible river, flowing steadily and relentlessly, carrying moments like leaves caught in its current, some gently floating by while others are swept swiftly into the vastness of memory. |
| Time is a relentless river, flowing endlessly, carrying the moments of our lives downstream, never to be revisited. | |
| 0.864 | Time is a river, ceaselessly flowing, carrying us along its currents. We may try to navigate its waters, steering toward calm eddies or bracing against its rapids, but we cannot stop its movement. Moments drift past like leaves, some lingering briefly on the surface before vanishing into the distance, while others sink into the depths, remembered only as faint ripples. Though we cannot hold the river, we can choose how we move within it—whether we struggle against the tide or learn to glide with its rhythm. |
| Time is a river, constantly flowing forward, carving through the landscape of our lives with relentless determination. It shapes the banks of our memories, deposits the sediment of our experiences, and carves out the canyons of our milestones. Some stretches are serene, meandering slowly through peaceful moments, allowing us to savour each droplet of existence. Other parts rush headlong, thunderous and unstoppable, carrying us along in its powerful current, challenging us to keep our footing on the slippery rocks of the present. In its journey, time erodes the impermanent and smooths the rough edges of what remains, creating the valleys and mountains of our history. We, like leaves or debris caught in its flow, must navigate its inexorable course, occasionally trying to swim against the tide, yet ultimately carried along by its unstoppable momentum. |
Table 16: Examples of semantically similar response pairs for the query “Write a metaphor involving time.” and their corresponding sentence similarity scores.
| Model | Prompt | Response |
| Llama-3.1-8B-Instruct | Write a metaphor involving time. | Time is a river that flows relentlessly, carrying memories and experiences along its currents, shaping the shores of our lives with every passing moment, and ultimately emptying into the vast ocean of eternity. |
| Create a metaphor that relates to time. | A river of moments flows through the expanse of existence, ever-changing and ever-present. Its waters, like grains of sand, slip through our fingers as the currents of time carry us along. Each eddy and ripple forms a distinct memory, while the relentless downstream drifts away with the passage of days. | |
| Compose a metaphor centered around time. | Time is a river , its waters constantly flowing, carrying moments past, present, and future along with them. It carves paths through the landscape of our lives, wearing down the stones of yesterday, sculpting the shores of today, and shaping the mountains of tomorrow. Its currents twist and turn, sometimes swift, sometimes still, but always moving forward, carrying us with it, ever-changing, ever-flowing. | |
| Come up with a metaphor that involves time. | Time is a river , flowing constantly and effortlessly, carrying moments, memories, and experiences with it. | |
| Generate a metaphor that features time. | Time is a river , flowing relentlessly forward, shaping the landscape of our lives as it twists and turns, carving out moments and memories, sometimes gently, sometimes violently, but always relentlessly moving, carrying us along with it, leaving behind a trail of past and present, as we drift toward the unknown shores of the future. | |
| Qwen1.5-110B-Chat | Create a metaphor that relates to time. | Time is like a river that flows constantly, carrying us along its current. Just as a river carves its path through the landscape, time shapes our lives and creates a unique journey for each of us. We can dip our toes in to pause and reflect, but we can never stop the flow or turn back to where we started. Every moment is a new downstream destination, and once it passes, it becomes a part of the vast ocean of memories. |
Table 17: Examples of repetitive responses produced by different models when prompted paraphrased versions of the same prompt: “Write a metaphor involving time.”
| Model | Prompt | Response |
| Llama-3.3-70B-Instruct | Write a paragraph about how the internet shaped society. | The internet has profoundly shaped society in countless ways, revolutionizing the way we communicate, access information, and interact with one another. […] |
| Compose a paragraph discussing the impact the internet has had on society. | The advent of the internet has profoundly impacted society, revolutionizing the way we communicate, access information, and conduct our daily lives. […] | |
| Draft a paragraph explaining the ways in which the internet has influenced society. | The advent of the internet has profoundly impacted society, transforming the way we communicate, access information, and interact with one another. […] | |
| Write a paragraph describing how society has been transformed by the internet. | The advent of the internet has revolutionized modern society in profound ways, transforming the way we communicate, access information, and interact with one another. […] | |
| Create a paragraph outlining the effects of the internet on society. | The internet has profoundly impacted society, bringing about numerous transformative effects that have reshaped the way we live, interact, and access information. […] | |
| Qwen2.5-72B-Instruct | Write a paragraph about how the internet shaped society. | The internet has profoundly reshaped society in numerous ways, transforming how we communicate, work, learn, and interact with the world. […] |
Table 18: Examples of repetitive responses produced by different models when prompted paraphrased versions of the same prompt: “Write a paragraph about how the internet shaped society.”
Figure 15: Responses to the query “Name one meaning of life.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( ) and temperature . We observe prominent clusters, indicating substantial overlap in the responses across many models.
Figure 16: Responses to the query “Provide a few sentences on Sisu Cinema Robotics.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( ) and temperature . We observe prominent clusters, indicating substantial overlap in the responses across many models.
Figure 17: Responses to the query “Write a short story about a colorful toad goes on an adventure.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( ) and temperature . We observe prominent clusters, indicating substantial overlap in the responses across many models.
Figure 18: Responses to the query “Generate a joke about electric vehicles.” clustered by applying PCA to reduce sentence embeddings to two dimensions. Each of the 25 models generates 50 responses using top- p sampling ( ) and temperature . We observe prominent clusters, indicating substantial overlap in the responses across many models.
| Method | LM Perplexities | Reward Model Scores | LM Judge Scores | ||||||
| (full = .361) | (full = .330) | (full = .305) | |||||||
| Top % of Sim. Quality | 80% | 60% | 40% | 80% | 60% | 40% | 80% | 60% | 40% |
| Optimized Sliding (L) | .365 | .412 | .341 | .316 | .312 | .300 | .268 | .248 | .226 |
| Centroid-Based (L) | .347 | .372 | .357 | .300 | .297 | .278 | .259 | .230 | .206 |
| Distance-Based (L) | .346 | .372 | .357 | .300 | .297 | .278 | .259 | .230 | .206 |
| Tukey (L) | .365 | .412 | .341 | .316 | .312 | .300 | .268 | .248 | .226 |
Table 19: Spearman correlation coefficients of various similarity-based metrics across subsets of the top-% of most similar-quality examples, evaluated between average human scores and LM perplexities, reward model scores, and LM judge scores. See §Appendix D.3 for details on all similar subset selection methods. L denotes methods applied to responses from the same query, while G denotes methods applied to the global pool of responses across all queries.
| Method | LM Perplexities | Reward Model Scores | LM Judge Scores | |||||||||
| (full = .361) | (full = .330) | (full = .305) | ||||||||||
| Top N of Disagreed | 120 | 90 | 60 | 30 | 120 | 90 | 60 | 30 | 120 | 90 | 60 | 30 |
| Entropy | .170 | .045 | -.030 | -.108 | .292 | .228 | .175 | -.073 | .287 | .254 | .276 | .070 |
| Entropy Grouped | .137 | .004 | .049 | -.160 | .293 | .246 | .081 | .153 | .247 | .227 | .121 | .139 |
| Gini Impurity | .071 | .015 | .029 | -.108 | .331 | .284 | .129 | -.043 | .299 | .278 | .216 | .162 |
| Pairwise Disagreement | .063 | .015 | .029 | .038 | .337 | .284 | .129 | -.012 | .295 | .278 | .216 | .103 |
Table 20: Spearman correlation coefficients of various disagreement-based metrics across subsets of the top- most disagreed examples, evaluated between average human scores and LM perplexities, reward model scores, and LM judge scores. See §Appendix D.4 for details on all similar subset selection methods.
| Prescreening Criterion | Requirement |
| Primary Language | English |
| First Language | English |
| Fluent Languages | English |
| Highest Education Level Completed | High school diploma/A-levels, Technical/community college, Undergraduate degree (BA/BSc/other), Graduate degree (MA/MSc/MPhil/other), Doctorate degree (PhD/other) |
| Number of Previous Submissions | 100–10,000 |
| Approval Rate | 99–100% |
Table 21: Prescreening criteria for annotator selection at Prolific.
| Demographic Dimension | Summary |
| Total Annotators | 2,296 |
| Disclosed Demographic Info | 95.0% (2,181 / 2,296) |
| Nationality | 38 unique; Top 3: United States (43.5%), United Kingdom (29.3%), Canada (12.5%) |
| Age | Mean = 39.9, SD = 13.4 |
| Ethnicity | White (68.6%), Black (13.8%), Asian (9.3%), Mixed (5.3%), Other (2.6%) |
| Education | Undergraduate degree (40.8%), Graduate degree (21.4%), High school diploma (20.4%), Technical/Community college (13.9%), Doctorate degree (3.5%) |
Table 22: Demographic summary of all annotators.
Figure 19: Screenshot of the human annotation interface for the absolute rating task (screen 1).
Figure 20: Screenshot of the human annotation interface for the absolute rating task (screen 2).
Figure 21: Screenshot of the human annotation interface for the pairwise preference rating task (screen 1).
Figure 22: Screenshot of the human annotation interface for the pairwise preference rating task (screen 2).
Figure 23: Prompt for the Overall Quality LM Judge
Figure 24: Prompt for the Helpfulness/Harmlessness/Honest LM Judge
| Type | Model Name | Full | ||||||
| N | 750 | 745 | 739 | 731 | 718 | 695 | 598 | |
| Language Models (Perplexities) | meta-llama/Llama-3.1-8B-Instruct | 0.364 | 0.353 | 0.345 | 0.339 | 0.334 | 0.325 | 0.266 |
| meta-llama/Llama-3.1-70B-Instruct | 0.370 | 0.359 | 0.350 | 0.344 | 0.340 | 0.330 | 0.269 | |
| meta-llama/Llama-3.2-1B-Instruct | 0.361 | 0.350 | 0.342 | 0.336 | 0.331 | 0.322 | 0.261 | |
| meta-llama/Llama-3.2-3B-Instruct | 0.365 | 0.354 | 0.345 | 0.339 | 0.334 | 0.325 | 0.265 | |
| meta-llama/Llama-3.3-70B-Instruct | 0.354 | 0.344 | 0.335 | 0.332 | 0.326 | 0.315 | 0.256 |
Table 23: Absolute rating model calibration analysis is conducted on a subset of instances with similar human scores, excluding outliers based on differences determined by Tukey’s fence values. Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while indicates the multiplier used in Tukey’s method to define the outlier range beyond the interquartile range.
| Type | Model Name | Full | ||||||||
| N | 750 | 123 | 105 | 90 | 75 | 63 | 45 | 30 | 17 | |
| Language Models (Perplexities) | meta-llama/Llama-3.1-8B-Instruct | 0.364 | 0.188 | 0.068 | 0.068 | 0.079 | 0.012 | 0.052 | 0.010 | 0.131 |
| meta-llama/Llama-3.1-70B-Instruct | 0.370 | 0.194 | 0.079 | 0.082 | 0.087 | 0.025 | 0.063 | 0.002 | 0.151 | |
| meta-llama/Llama-3.2-1B-Instruct | 0.361 | 0.187 | 0.068 | 0.069 | 0.080 | 0.000 | 0.028 | 0.007 | 0.186 | |
| meta-llama/Llama-3.2-3B-Instruct | 0.365 | 0.184 | 0.062 | 0.059 | 0.060 | -0.002 | 0.041 | 0.023 | 0.163 | |
| meta-llama/Llama-3.3-70B-Instruct | 0.354 | 0.172 | 0.055 | 0.061 | 0.065 | 0.006 | 0.037 | 0.057 | 0.254 |
Table 24: Absolute rating model calibration analysis is conducted on a subset of instances with high human disagreement . Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while specifies the top % of instances with the highest disagreement among human scores.
| Type | Model Name | Full | ||||||||
| N | 500 | 475 | 450 | 425 | 400 | 375 | 350 | 325 | 300 | |
| Language Models (Perplexities) | meta-llama/Llama-3.1-8B-Instruct | 0.431 | 0.410 | 0.384 | 0.365 | 0.369 | 0.337 | 0.311 | 0.277 | 0.246 |
| meta-llama/Llama-3.1-70B-Instruct | 0.428 | 0.419 | 0.397 | 0.377 | 0.379 | 0.348 | 0.332 | 0.289 | 0.257 | |
| meta-llama/Llama-3.2-1B-Instruct | 0.417 | 0.406 | 0.377 | 0.355 | 0.356 | 0.324 | 0.312 | 0.276 | 0.235 | |
| meta-llama/Llama-3.2-3B-Instruct | 0.418 | 0.407 | 0.375 | 0.357 | 0.359 | 0.326 | 0.301 | 0.265 | 0.233 | |
| meta-llama/Llama-3.3-70B-Instruct | 0.381 | 0.384 | 0.362 | 0.344 | 0.336 | 0.315 | 0.293 | 0.263 | 0.247 |
Table 25: Pairwise preference rating model calibration analysis is conducted on a subset of instances with similar human scores. Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while specifies the top % of instances with the highest similarity in human preference scores.
| Type | Model Name | Full | ||||||||
| N | 500 | 475 | 450 | 425 | 400 | 375 | 350 | 325 | 300 | |
| Language Models (Perplexities) | Model Names | full | 95.00 | 90.00 | 85.00 | 80.00 | 75.00 | 70.00 | 65.00 | 60.00 |
| meta-llama/Llama-3.1-8B-Instruct | 0.431 | 0.398 | 0.353 | 0.314 | 0.300 | 0.283 | 0.300 | 0.271 | 0.250 | |
| meta-llama/Llama-3.1-70B-Instruct | 0.428 | 0.405 | 0.367 | 0.322 | 0.310 | 0.293 | 0.308 | 0.285 | 0.261 | |
| meta-llama/Llama-3.2-1B-Instruct | 0.417 | 0.383 | 0.335 | 0.303 | 0.279 | 0.265 | 0.279 | 0.258 | 0.229 | |
| meta-llama/Llama-3.2-3B-Instruct | 0.418 | 0.390 | 0.347 | 0.308 | 0.290 | 0.272 | 0.287 | 0.265 | 0.237 |
Table 26: Pairwise preference rating model calibration analysis is conducted on a subset of instances with high disagreement between different human annotators. Spearman’s correlation coefficients are computed between human-annotated scores and model-generated scores across three categories: LMs , LM judges , and reward models , evaluated on various sets of model responses. Full denotes the complete set of responses, while specifies the top % of instances with the highest disagreement in human preference scores.