Revisiting LLM Homogeneity Through a Multidimensional Lens

Ziliang Qiu; Renfen Hu; Yi Ding; Ted Underwood

cs.CLNov 1, 2026

Revisiting LLM Homogeneity Through a Multidimensional Lens

Authors: Ziliang Qiu, Renfen Hu, Yi Ding, Ted Underwood

Organizations: University of Illinois Urbana-Champaign · Beijing Normal University · Purdue University

Abstract

Concerns about LLM homogeneity are typically quantified with embedding similarity, which reduces two texts to a single score. We revisit homogeneity with 25 interpretable metrics spanning five dimensions: semantic, lexical, syntactic, structural, and stylistic & presentational. A human study shows that perceived similarity depends on both content (what responses say) and form (how they say it), yet embedding similarity correlates less with human meaning judgments than sentence-level matching ( $ρ=0.417$ vs. $0.606$ ) and not significantly with form judgments ( $ρ=0.140$ ). Across 18 contemporary LLMs, models say similar things in different words: they share global semantics and broad linguistic profiles but remain distinct in lexical characteristics, structural patterns, and interactional conventions. Across releases from 2024 to 2026, models become more like themselves but not more like one another: among significantly changing metrics, intra-model distances decrease for 17 of 22, and inter-model distances increase for 14 of 21. Across two post-training pipelines, model-specific conventions emerge during SFT and are selectively reshaped by preference optimization. These results suggest that homogeneity is multidimensional and evolving, and that embedding similarity alone can overstate cross-model homogeneity while concealing diverging intra- and inter-model trends.

Figures & tables

Figure 1: An example pair from Infinity-Chat ( Jiang et al., 2025 ) showing embedding similarity (Qwen3-Embedding-8B) and human similarity ratings (§ 4.1 ). The responses differ in definition and structure. Despite high embedding similarity, annotators rate the pair 2/5 overall, 3/5 for meaning, and 1/5 for form. See Figure 5 for the full example.

	Aspect	Metrics
	Semantic
	Global semantic similarity	Embedding Similarity
	Sentence-level correspondence	InfoMatch (Sent)
	Lexical overlap	BLEU, ROUGE-L
	Lexical
	Lexical characteristics	Lexical Diversity (MTLD), Word Frequency (Zipf)

Table 1: Dimensions and aspects of textual similarity, with their corresponding metrics. Detailed metric definitions and computation methods are provided in Table 13 and Appendix A .

Figure 2: Spearman correlations of similarity metrics with embedding similarity (first column; $n=500$ pairs) and with human judgments of overall, meaning, and form similarity (remaining columns; $n=90$ pairs; ratings averaged across three annotators). Exact values are reported in Table 12 (human judgments) and Table 14 (embedding similarity).

	Distance
Metric	Intra	Inter	Rand	$N$
Weak separation ( $N>0.7$ )
Embedding Similarity	.101	.176	.895	.905
InfoMatch (Sent)	.263	.321	.559	.805
Emotion Profile	.062	.115	.325	.797
Concreteness	.045	.084	.218	.776

Table 2: Intra-model, inter-model, and random-pair distances across 25 textual metrics for 18 models (153 unordered model pairs). All values are expressed as distances, with semantic similarities

S

converted to

1-S

N

(Eq. 1 ) denotes normalized inter-model similarity. Lower

N

indicates greater cross-model separation. Blue and orange shading mark the five metrics with the highest observed correlations with human meaning and form similarity judgments, respectively (§ 4.1 ). Bars visualize

N

on a 0–1 scale.

Figure 3: Inter- and intra-model distances from 2024 to 2026 for three representative metrics. On all three, models become more like themselves (decreasing intra-model distance) but less like one another (increasing inter-model distance). See Figure 8 for all metrics.

Metric	$\Delta D_{\mathrm{inter}}$ (2024 $\to$ 2026)
Human-Salient Meaning Features
Embedding Similarity	$\blacktriangledown$ $-$ 10.0%
InfoMatch	$\blacktriangle$ +9.6%
Punctuation Profile †	$\blacktriangle$ +7.1%
ROUGE-L	$\blacktriangle$ +3.2%
BLEU	$\blacktriangle$ +1.2%

Table 3: Changes in inter-model distance from 2024 to 2026. Human-salient meaning and form features denote the five metrics with the highest observed correlations with human meaning and form judgments (§ 4.1 ).

\Delta D_{\mathrm{inter}}

is the relative change from 2024:

\blacktriangle

means models diverge and

\blacktriangledown

means they converge. Grey entries ( † ) have 95% prompt-level bootstrap CIs (10,000 resamples) that include zero.

Figure 4: Post-training narrows semantic and grammatical distances while interactional and formatting conventions diverge. Training-stage trends for six representative metrics in OLMo 3 and Tülu 3. Dashed lines show adoption rates for Opening Template/Header Use and intra-model distances otherwise; black lines show mean inter-model distances with 95% bootstrap CIs shaded. Lower distances indicate greater similarity. Base (pure/form) uses three-shot demonstrations without/with formatting markers. See Figure 9 for all metrics.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

View	Processing	Metrics	Rationale
Raw response	No preprocessing; the complete model output is retained	Embedding Similarity; Response Length	These metrics are explicitly defined over the response as originally produced
Semantic-content view	Markdown syntax and URLs are removed; fenced code blocks are excluded; inline-code content and textual table-cell content are retained	BLEU; ROUGE-L	Preserves linguistic content while reducing overlap caused by markup or large code blocks
Sentence-semantic view	Uses semantic-content blocks, additionally excludes tables, and segments each remaining block into sentences independently	InfoMatch (Sent)	Sentence-level correspondence requires natural sentence units; table rows can otherwise create artificial best matches
Prose view	Markdown syntax, URLs, tables, fenced code blocks, and inline code are excluded; natural-language textual content is retained	MTLD; Word Frequency; POS Distribution; Dependency Distribution; Function Word Profile/Density; Readability; Concreteness; Emotion Profile/Intensity; Punctuation Profile/Density	Restricts linguistic and stylistic measurements to natural-language prose and avoids contamination from code or table syntax
Structural view	Fenced code blocks are collapsed to single structural blocks and horizontal rules are removed; the remaining layout is preserved	Paragraph Density	Preserves document organization while preventing blank lines inside code blocks from creating artificial paragraphs
Markdown structure	The original Markdown structure is parsed directly; code regions are excluded when identifying additional nonstandard bullet markers	Header Use; List Use/Density; Markdown Emphasis Use/Density	These metrics depend on explicit presentational structure that is removed from the prose representation

Appendix

Table 4: Metric-specific response representations used during feature extraction. Each view retains information relevant to its associated metrics while removing content that could introduce measurement artifacts.

Output type	Example metric	Similarity
Pairwise	Embedding cosine	Native score
Scalar	Response Length	$-\|x_{i}-x_{j}\|$
Distributional	POS Distribution	$-\mathrm{JSD}(x_{i},x_{j})$
Binary	Header Use	$\mathbb{I}[x_{i}=x_{j}]$

Appendix

Table 5: Pairwise similarity computation by metric output type, with representative examples.

Rating	Score
Infinity-Chat (response quality)
Absolute, Response 1 (1–5)	3.20 $\pm$ 1.26
Absolute, Response 2 (1–5)	3.92 $\pm$ 0.74
Pairwise ( $-2$ to $2$ )	$-1.00$
Our study (response similarity)
Overall (1–5)	2 (3, 1, 2)

Appendix

Table 6: Human ratings and Qwen3-Embedding-8B similarity for the motivating example. Infinity-Chat ratings come from 25 annotators and assess response quality; absolute ratings are reported as mean

\pm

standard deviation and the pairwise rating as the mean. Our study collects similarity ratings from three annotators, reported as the mean followed by individual ratings.

Type	Per annotator	Source	Role
Main	33 or 34	90 sampled pairs, each seen by 3 annotators	Primary judgments
Halo	3 (shared)	Lowest-quartile similarity pairs outside the main set, with all form features closely matched	Testing whether surface-form similarity leads to inflated similarity ratings
Repeat	3	Annotator’s own main pairs, A/B reversed	Within-annotator consistency
Catch	2	Randomly sampled responses, identical on both sides ( $r_{A}=r_{B}$ )	Attention check
Total	41 or 42

Appendix

Table 7: Item types in the human annotation study.

Panel of Figure 6 — Figure 6: Annotation interface on Qualtrics. Each item shows the question and two responses side by side, followed by a five-point scale with verbal labels at points 1, 3, and 5. In Round 2, annotators give separate meaning and form ratings on the same page.

Pair	Overall	Meaning	Form
Lowest-quartile pairs (mean)	2.13	2.62	2.72
H1	2.13	2.00	4.00
H2	2.50	1.75	4.13
H3	2.63	2.50	3.63

Appendix

Table 8: Mean human ratings for the three halo pairs H1–H3 (

n=8

annotators per pair), with ordinary pairs from the lowest semantic-similarity quartile shown as a reference.

Model	Series	Stage	Ref.
API-served
deepseek-v4-flash-0423	deepseek-v4	-	DeepSeek-AI (2026)
gemini-3.1-flash-lite	gemini-3.1	-	Google DeepMind (2026)
gpt-5.4-nano	gpt-5.4	-	OpenAI (2026)
Gemma
gemma-2-9b-it	gemma-2	Instruct	Gemma Team et al. (2024)

Appendix

Table 9: Models evaluated in this work, grouped by series and training stage. Base : pretrained checkpoint; SFT : supervised fine-tuning; DPO : direct preference optimization; RLVR : reinforcement learning with verifiable rewards; Instruct : official instruction-tuned release; Thinking : release with explicit reasoning traces; API-served : models accessed through APIs. † Models supporting reasoning, evaluated in non-thinking mode. ‡ Only a DPO checkpoint has been released for this series, so we use it for the temporal comparison.

ID	Question / Prompt
Creative Content Generation
1	Create a character named Bill Honda.
2	Show me a joke?
3	Write about puns with the word “solana” and animal names.
4	Write a poem about a man.
5	I once got fired from a keyboard factory for not putting in enough shifts. Make up a similar joke about ships.

Appendix

Table 10: Example questions by category. Questions are sampled from Infinity-Chat ( Jiang et al., 2025 ) ; some are edited to remove stylistic constraints and format requirements.

Model	Filter rate (%)
Cross-model & temporal (18 models, 279 prompts)
gemma-2-9b-it	3.46
gemma-3-12b-it	0.93
gemma-4-12B-it †	0.91
OLMo-2-1124-7B-Instruct	2.71
Olmo-3-7B-Instruct	1.53

Appendix

Table 11: Percentage of responses discarded by the filter, grouped by analysis. All responses are sampled with top-

p

sampling. For base models, pure denotes few-shot prompting without formatting markers and form denotes few-shot prompting with formatting markers. Group headers list the model count and the number of prompts (of 300) with

\geq

30 valid post-filter responses for all models. † Hybrid-reasoning model evaluated in non-thinking mode.

Figure 7: Shared semantics with divergent wording holds across question types. Inter-model similarity by question type for 18 models; bars show the mean over all 153 model pairs. Questions with multiple labels count toward each assigned category. Upper-right values give each metric’s mean over all 279 questions. Abbreviations: Perspectives/Styles = Alternative Perspectives/Styles; Ideation = Brainstorming and Ideation; Creative = Creative Content Generation; Info seeking = Information-Seeking; Open ended = Open-Endedness.

Metric	Overall	Meaning	Form
Semantic
Embedding Similarity	$0.400$ ∗∗∗ $[0.209,0.561]$	$0.417$ ∗∗∗ $[0.220,0.586]$	$0.140$ $[-0.073,0.340]$
InfoMatch (Sent)	$0.593$ ∗∗∗ $[0.450,0.705]$	$\mathbf{0.606}$ ∗∗∗ $[0.462,0.722]$	$0.263$ ∗ $[0.059,0.448]$
BLEU	$\mathbf{0.608}$ ∗∗∗ $[0.461,0.722]$	$0.538$ ∗∗∗ $[0.358,0.688]$	$0.325$ ∗∗ $[0.104,0.514]$
ROUGE-L	$0.581$ ∗∗∗ $[0.422,0.707]$	$0.525$ ∗∗∗ $[0.345,0.670]$	$0.260$ ∗ $[0.035,0.460]$
Lexical

Appendix

Table 12: Spearman correlations between each metric and human ratings (

n=90

response pairs). Smaller gray values show 95% percentile bootstrap CIs (10,000 resamples of 90 pairs with replacement). Blue and orange cells mark the five metrics with the highest observed correlations with human meaning and form judgments, respectively. These correspond to the highlighted metrics in Table 2 and the groups in Table 3 . Bold marks each column’s highest correlation. All metrics are sign-aligned. Stars denote unadjusted Spearman

p

-values:

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

. StyleDistance is a reference metric outside our inventory. ‡ MTLD uses

n=75

valid pairs, with undefined values excluded in each bootstrap resample.

Dimension	Metric	Description	Output	Similarity	Ref.
Semantic	Embedding Similarity	Cosine similarity between Qwen3-Embedding-8B representations of the complete responses; used as the primary semantic baseline	pairwise	native	Zhang et al. (2025a)
	InfoMatch (Sent)	Sentence-level semantic correspondence computed by bidirectional best-match cosine matching between Qwen3-Embedding-8B sentence representations, with the two directional scores combined using their harmonic mean; code blocks and tables are excluded	pairwise	native	Zhang et al. (2020) ; Zhang et al. (2025a)
	BLEU	Symmetric response-level BLEU using SacreBLEU sentence scoring with lowercased up-to-4-gram overlap, effective order, exponential smoothing, and brevity penalty; code blocks are excluded	pairwise	native	Papineni et al. (2002)
	ROUGE-L	F1 score based on the longest common subsequence of the normalized textual content; code blocks are excluded	pairwise	native	Lin (2004)
Lexical	Lexical Diversity (MTLD)	Length-robust measure of vocabulary diversity within a response, computed for responses containing at least 50 lexical tokens	scalar	$-\|\Delta\|$	McCarthy and Jarvis (2010)
Lexical	Word Frequency (Zipf)	Mean Zipf frequency of words in a response according to wordfreq ; out-of-vocabulary words receive frequency 0 and remain in the average	scalar	$-\|\Delta\|$	Speer (2022) ; van Heuven et al. (2014)

Appendix

Table 13: Metric inventory for the five dimensions of LLM response similarity, details can be found in Appendix A .

Figure 8: Inter- and intra-model distances across all 25 textual metrics from 2024 to 2026. Shaded bands indicate 95% bootstrap confidence intervals (10,000 prompt-level resamples, with prompts jointly resampled across models and years). Gray backgrounds indicate metrics with increasing inter-model and decreasing intra-model distances, and arrows indicate the direction of change from 2024 to 2026. Lower values indicate greater similarity.

Figure 9: Inter- and intra-model distances across all 25 textual metrics for OLMo 3 and Tülu 3 over post-training stages. Solid black lines denote inter-model distances; dashed orange and blue lines denote intra-model distances for OLMo 3 and Tülu 3, respectively. Shaded bands indicate 95% bootstrap confidence intervals (10,000 prompt-level resamples). Base ( pure / form ) denotes two alternative few-shot prompting conditions, and SFT, DPO, and RLVR denote successive post-training checkpoints. Lower values indicate greater similarity.

Metric	Qwen3	Jina	QZhou
Semantic
InfoMatch (Sent) †	0.646 ∗∗∗	0.725 ∗∗∗	0.599 ∗∗∗
BLEU	0.633 ∗∗∗	0.559 ∗∗∗	0.482 ∗∗∗
ROUGE-L	0.636 ∗∗∗	0.537 ∗∗∗	0.468 ∗∗∗
Lexical
Lexical Diversity (MTLD)	0.019	$-$ 0.053	$-$ 0.022

Appendix

Table 14: Spearman correlations between whole-response embedding similarity and each response-similarity metric on Infinity-Chat , using Qwen3-Embedding-8B, jina-embeddings-v5-text, and QZhou-Embedding (

n=500

response pairs). For each metric, pairs that do not meet its computation requirements are excluded.

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

. † InfoMatch is computed with the same embedding model as the corresponding column.

Homogenization in Language Models

See all 10 papers

Oct 12, 2026

The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?

Text generated by Large Language Models (LLMs) often remains limited in diversity compared to that produced by humans. Recent advances demonstrate the presence of inter- and intra-model collapse in open-ended generation of English text, raising concerns about the long-term homogenization of LLM, and more importantly, human thought. However, our understanding of whether this homogenization extends to languages beyond English is limited. In this work, we investigate the role of language on output diversity. Specifically, we aim to understand whether multilingual LLM-generated text is also homogeneous, and whether this homogeneity matches what we observe in humans. Across 23 open-weight models and 13 languages, we find that the artificial hivemind effect is pervasive but uneven in its extent. The intra-model effect holds in three different embedding spaces and under a lexical metric that uses no embedding model, so it is not an artifact of how diversity is measured. The size of the homogenization gap is associated with a model's competence in a language, measured on a held-out translation benchmark, robustly across embedding spaces. Thus, we demonstrate a performance-diversity tradeoff in multilingual LLMs.

Xaver Raphael Davey, Shubhra Mishra, Anton Adelöw +1

KTH Royal Institute of Technology

Output Diversity in Language Models Homogenization in Language Models Multilingual Language Models Multilingual Language Model Evaluation

Ask PDF

Mar 20, 2026

Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities

LLM assistant personalities play a critical role in user experience and perceived response quality. We present a large-scale experiment of frontier LLM personalities using external ELO-based traits scoring across 144 traits. We find that all models tested converge on a form of trait expression that is systematic, methodical, and analytical and suppress traits such as remorseful and sycophantic. Moreover, models tend to diverge more in their expression of middle-of-distribution traits such as poetic or playful, but even these so-called creative models tend to have more neutral identities. These similarities suggest an implicit emergence of a standard of optimal assistant behavior. In a landscape of varied training methods, character training, therefore, stands out for its uniformity, offering insight into a tacit consensus between model developers.

Avinash Krishna, Kalyana Chadalavada, Unso Eun Seo Jo

Independent · Anthropic · Cornell University

Personality Modeling in Language Models Behavioral Consistency of LLM Agents Normative Conformity in Language Models Language Model Post-Training

Ask PDF

Aug 13, 2026

Semantic Variability of Replies Across LLMs: Implications for Designing Conversation-Based Assessment

This study examines whether LLM-generated replies remain semantically consistent when the underlying LLM changes. Using messages from real collaborative conversations, we compared the semantic similarity of generated replies across LLMs under two conditions: with and without preceding chat history. Results show that model choice and conversational context both affect response similarity and alignment with human replies. These findings indicate that prompting and conversational context alone may not be sufficient to preserve response consistency across LLMs, highlighting the need for infrastructure and design strategies that can maintain stable and comparable responses amid the rapid and continuous evolution of LLMs.

Jiangang Hao

ETS Research Institute Princeton, New Jersey, USA

LLM Reliability Language Model Generation Evaluation Multi-Turn Dialogue Evaluation

Ask PDF

Abstract

Homogenization in Language Models

Explore similar work

The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?

Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities

Semantic Variability of Replies Across LLMs: Implications for Designing Conversation-Based Assessment