cs.CLNov 1, 2026
SaveRevisiting LLM Homogeneity Through a Multidimensional Lens
Organizations: University of Illinois Urbana-Champaign · Beijing Normal University · Purdue University
Abstract
Concerns about LLM homogeneity are typically quantified with embedding similarity, which reduces two texts to a single score. We revisit homogeneity with 25 interpretable metrics spanning five dimensions: semantic, lexical, syntactic, structural, and stylistic & presentational. A human study shows that perceived similarity depends on both content (what responses say) and form (how they say it), yet embedding similarity correlates less with human meaning judgments than sentence-level matching ( vs. ) and not significantly with form judgments (). Across 18 contemporary LLMs, models say similar things in different words: they share global semantics and broad linguistic profiles but remain distinct in lexical characteristics, structural patterns, and interactional conventions. Across releases from 2024 to 2026, models become more like themselves but not more like one another: among significantly changing metrics, intra-model distances decrease for 17 of 22, and inter-model distances increase for 14 of 21. Across two post-training pipelines, model-specific conventions emerge during SFT and are selectively reshaped by preference optimization. These results suggest that homogeneity is multidimensional and evolving, and that embedding similarity alone can overstate cross-model homogeneity while concealing diverging intra- and inter-model trends.
Figures & tables
Figure 1: An example pair from Infinity-Chat ( Jiang et al., 2025 ) showing embedding similarity (Qwen3-Embedding-8B) and human similarity ratings (§ 4.1 ). The responses differ in definition and structure. Despite high embedding similarity, annotators rate the pair 2/5 overall, 3/5 for meaning, and 1/5 for form. See Figure 5 for the full example.
| Aspect | Metrics | |
| Semantic | ||
| Global semantic similarity | Embedding Similarity | |
| Sentence-level correspondence | InfoMatch (Sent) | |
| Lexical overlap | BLEU, ROUGE-L | |
| Lexical | ||
| Lexical characteristics | Lexical Diversity (MTLD), Word Frequency (Zipf) | |
Table 1: Dimensions and aspects of textual similarity, with their corresponding metrics. Detailed metric definitions and computation methods are provided in Table 13 and Appendix A .
Figure 2: Spearman correlations of similarity metrics with embedding similarity (first column; pairs) and with human judgments of overall, meaning, and form similarity (remaining columns; pairs; ratings averaged across three annotators). Exact values are reported in Table 12 (human judgments) and Table 14 (embedding similarity).
| Distance | ||||
| Metric | Intra | Inter | Rand | |
| Weak separation ( ) | ||||
| Embedding Similarity | .101 | .176 | .895 | .905 |
| InfoMatch (Sent) | .263 | .321 | .559 | .805 |
| Emotion Profile | .062 | .115 | .325 | .797 |
| Concreteness | .045 | .084 | .218 | .776 |
Table 2: Intra-model, inter-model, and random-pair distances across 25 textual metrics for 18 models (153 unordered model pairs). All values are expressed as distances, with semantic similarities converted to . (Eq. 1 ) denotes normalized inter-model similarity. Lower indicates greater cross-model separation. Blue and orange shading mark the five metrics with the highest observed correlations with human meaning and form similarity judgments, respectively (§ 4.1 ). Bars visualize on a 0–1 scale.
Figure 3: Inter- and intra-model distances from 2024 to 2026 for three representative metrics. On all three, models become more like themselves (decreasing intra-model distance) but less like one another (increasing inter-model distance). See Figure 8 for all metrics.
| Metric | (2024 2026) |
| Human-Salient Meaning Features | |
| Embedding Similarity | 10.0% |
| InfoMatch | +9.6% |
| Punctuation Profile † | +7.1% |
| ROUGE-L | +3.2% |
| BLEU | +1.2% |
Table 3: Changes in inter-model distance from 2024 to 2026. Human-salient meaning and form features denote the five metrics with the highest observed correlations with human meaning and form judgments (§ 4.1 ). is the relative change from 2024: means models diverge and means they converge. Grey entries ( † ) have 95% prompt-level bootstrap CIs (10,000 resamples) that include zero.
Figure 4: Post-training narrows semantic and grammatical distances while interactional and formatting conventions diverge. Training-stage trends for six representative metrics in OLMo 3 and Tülu 3. Dashed lines show adoption rates for Opening Template/Header Use and intra-model distances otherwise; black lines show mean inter-model distances with 95% bootstrap CIs shaded. Lower distances indicate greater similarity. Base (pure/form) uses three-shot demonstrations without/with formatting markers. See Figure 9 for all metrics.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| View | Processing | Metrics | Rationale |
| Raw response | No preprocessing; the complete model output is retained | Embedding Similarity; Response Length | These metrics are explicitly defined over the response as originally produced |
| Semantic-content view | Markdown syntax and URLs are removed; fenced code blocks are excluded; inline-code content and textual table-cell content are retained | BLEU; ROUGE-L | Preserves linguistic content while reducing overlap caused by markup or large code blocks |
| Sentence-semantic view | Uses semantic-content blocks, additionally excludes tables, and segments each remaining block into sentences independently | InfoMatch (Sent) | Sentence-level correspondence requires natural sentence units; table rows can otherwise create artificial best matches |
| Prose view | Markdown syntax, URLs, tables, fenced code blocks, and inline code are excluded; natural-language textual content is retained | MTLD; Word Frequency; POS Distribution; Dependency Distribution; Function Word Profile/Density; Readability; Concreteness; Emotion Profile/Intensity; Punctuation Profile/Density | Restricts linguistic and stylistic measurements to natural-language prose and avoids contamination from code or table syntax |
| Structural view | Fenced code blocks are collapsed to single structural blocks and horizontal rules are removed; the remaining layout is preserved | Paragraph Density | Preserves document organization while preventing blank lines inside code blocks from creating artificial paragraphs |
| Markdown structure | The original Markdown structure is parsed directly; code regions are excluded when identifying additional nonstandard bullet markers | Header Use; List Use/Density; Markdown Emphasis Use/Density | These metrics depend on explicit presentational structure that is removed from the prose representation |
Table 4: Metric-specific response representations used during feature extraction. Each view retains information relevant to its associated metrics while removing content that could introduce measurement artifacts.
| Output type | Example metric | Similarity |
| Pairwise | Embedding cosine | Native score |
| Scalar | Response Length | |
| Distributional | POS Distribution | |
| Binary | Header Use |
Table 5: Pairwise similarity computation by metric output type, with representative examples.
| Rating | Score |
| Infinity-Chat (response quality) | |
| Absolute, Response 1 (1–5) | 3.20 1.26 |
| Absolute, Response 2 (1–5) | 3.92 0.74 |
| Pairwise ( to ) | |
| Our study (response similarity) | |
| Overall (1–5) | 2 (3, 1, 2) |
Table 6: Human ratings and Qwen3-Embedding-8B similarity for the motivating example. Infinity-Chat ratings come from 25 annotators and assess response quality; absolute ratings are reported as mean standard deviation and the pairwise rating as the mean. Our study collects similarity ratings from three annotators, reported as the mean followed by individual ratings.
| Type | Per annotator | Source | Role |
| Main | 33 or 34 | 90 sampled pairs, each seen by 3 annotators | Primary judgments |
| Halo | 3 (shared) | Lowest-quartile similarity pairs outside the main set, with all form features closely matched | Testing whether surface-form similarity leads to inflated similarity ratings |
| Repeat | 3 | Annotator’s own main pairs, A/B reversed | Within-annotator consistency |
| Catch | 2 | Randomly sampled responses, identical on both sides ( ) | Attention check |
| Total | 41 or 42 |
Table 7: Item types in the human annotation study.
Figure 6: Annotation interface on Qualtrics. Each item shows the question and two responses side by side, followed by a five-point scale with verbal labels at points 1, 3, and 5. In Round 2, annotators give separate meaning and form ratings on the same page.
| Pair | Overall | Meaning | Form |
| Lowest-quartile pairs (mean) | 2.13 | 2.62 | 2.72 |
| H1 | 2.13 | 2.00 | 4.00 |
| H2 | 2.50 | 1.75 | 4.13 |
| H3 | 2.63 | 2.50 | 3.63 |
Table 8: Mean human ratings for the three halo pairs H1–H3 ( annotators per pair), with ordinary pairs from the lowest semantic-similarity quartile shown as a reference.
| Model | Series | Stage | Ref. |
| API-served | |||
| deepseek-v4-flash-0423 | deepseek-v4 | - | DeepSeek-AI (2026) |
| gemini-3.1-flash-lite | gemini-3.1 | - | Google DeepMind (2026) |
| gpt-5.4-nano | gpt-5.4 | - | OpenAI (2026) |
| Gemma | |||
| gemma-2-9b-it | gemma-2 | Instruct | Gemma Team et al. (2024) |
Table 9: Models evaluated in this work, grouped by series and training stage. Base : pretrained checkpoint; SFT : supervised fine-tuning; DPO : direct preference optimization; RLVR : reinforcement learning with verifiable rewards; Instruct : official instruction-tuned release; Thinking : release with explicit reasoning traces; API-served : models accessed through APIs. † Models supporting reasoning, evaluated in non-thinking mode. ‡ Only a DPO checkpoint has been released for this series, so we use it for the temporal comparison.
| ID | Question / Prompt |
| Creative Content Generation | |
| 1 | Create a character named Bill Honda. |
| 2 | Show me a joke? |
| 3 | Write about puns with the word “solana” and animal names. |
| 4 | Write a poem about a man. |
| 5 | I once got fired from a keyboard factory for not putting in enough shifts. Make up a similar joke about ships. |
Table 10: Example questions by category. Questions are sampled from Infinity-Chat ( Jiang et al., 2025 ) ; some are edited to remove stylistic constraints and format requirements.
| Model | Filter rate (%) |
| Cross-model & temporal (18 models, 279 prompts) | |
| gemma-2-9b-it | 3.46 |
| gemma-3-12b-it | 0.93 |
| gemma-4-12B-it † | 0.91 |
| OLMo-2-1124-7B-Instruct | 2.71 |
| Olmo-3-7B-Instruct | 1.53 |
Table 11: Percentage of responses discarded by the filter, grouped by analysis. All responses are sampled with top- sampling. For base models, pure denotes few-shot prompting without formatting markers and form denotes few-shot prompting with formatting markers. Group headers list the model count and the number of prompts (of 300) with 30 valid post-filter responses for all models. † Hybrid-reasoning model evaluated in non-thinking mode.
Figure 7: Shared semantics with divergent wording holds across question types. Inter-model similarity by question type for 18 models; bars show the mean over all 153 model pairs. Questions with multiple labels count toward each assigned category. Upper-right values give each metric’s mean over all 279 questions. Abbreviations: Perspectives/Styles = Alternative Perspectives/Styles; Ideation = Brainstorming and Ideation; Creative = Creative Content Generation; Info seeking = Information-Seeking; Open ended = Open-Endedness.
| Metric | Overall | Meaning | Form |
| Semantic | |||
| Embedding Similarity | ∗∗∗ | ∗∗∗ | |
| InfoMatch (Sent) | ∗∗∗ | ∗∗∗ | ∗ |
| BLEU | ∗∗∗ | ∗∗∗ | ∗∗ |
| ROUGE-L | ∗∗∗ | ∗∗∗ | ∗ |
| Lexical | |||
Table 12: Spearman correlations between each metric and human ratings ( response pairs). Smaller gray values show 95% percentile bootstrap CIs (10,000 resamples of 90 pairs with replacement). Blue and orange cells mark the five metrics with the highest observed correlations with human meaning and form judgments, respectively. These correspond to the highlighted metrics in Table 2 and the groups in Table 3 . Bold marks each column’s highest correlation. All metrics are sign-aligned. Stars denote unadjusted Spearman -values: , , . StyleDistance is a reference metric outside our inventory. ‡ MTLD uses valid pairs, with undefined values excluded in each bootstrap resample.
| Dimension | Metric | Description | Output | Similarity | Ref. |
| Semantic | Embedding Similarity | Cosine similarity between Qwen3-Embedding-8B representations of the complete responses; used as the primary semantic baseline | pairwise | native | Zhang et al. (2025a) |
| InfoMatch (Sent) | Sentence-level semantic correspondence computed by bidirectional best-match cosine matching between Qwen3-Embedding-8B sentence representations, with the two directional scores combined using their harmonic mean; code blocks and tables are excluded | pairwise | native | Zhang et al. (2020) ; Zhang et al. (2025a) | |
| BLEU | Symmetric response-level BLEU using SacreBLEU sentence scoring with lowercased up-to-4-gram overlap, effective order, exponential smoothing, and brevity penalty; code blocks are excluded | pairwise | native | Papineni et al. (2002) | |
| ROUGE-L | F1 score based on the longest common subsequence of the normalized textual content; code blocks are excluded | pairwise | native | Lin (2004) | |
| Lexical | Lexical Diversity (MTLD) | Length-robust measure of vocabulary diversity within a response, computed for responses containing at least 50 lexical tokens | scalar | McCarthy and Jarvis (2010) | |
| Word Frequency (Zipf) | Mean Zipf frequency of words in a response according to wordfreq ; out-of-vocabulary words receive frequency 0 and remain in the average | scalar | Speer (2022) ; van Heuven et al. (2014) |
Table 13: Metric inventory for the five dimensions of LLM response similarity, details can be found in Appendix A .
Figure 8: Inter- and intra-model distances across all 25 textual metrics from 2024 to 2026. Shaded bands indicate 95% bootstrap confidence intervals (10,000 prompt-level resamples, with prompts jointly resampled across models and years). Gray backgrounds indicate metrics with increasing inter-model and decreasing intra-model distances, and arrows indicate the direction of change from 2024 to 2026. Lower values indicate greater similarity.
Figure 9: Inter- and intra-model distances across all 25 textual metrics for OLMo 3 and Tülu 3 over post-training stages. Solid black lines denote inter-model distances; dashed orange and blue lines denote intra-model distances for OLMo 3 and Tülu 3, respectively. Shaded bands indicate 95% bootstrap confidence intervals (10,000 prompt-level resamples). Base ( pure / form ) denotes two alternative few-shot prompting conditions, and SFT, DPO, and RLVR denote successive post-training checkpoints. Lower values indicate greater similarity.
| Metric | Qwen3 | Jina | QZhou |
| Semantic | |||
| InfoMatch (Sent) † | 0.646 ∗∗∗ | 0.725 ∗∗∗ | 0.599 ∗∗∗ |
| BLEU | 0.633 ∗∗∗ | 0.559 ∗∗∗ | 0.482 ∗∗∗ |
| ROUGE-L | 0.636 ∗∗∗ | 0.537 ∗∗∗ | 0.468 ∗∗∗ |
| Lexical | |||
| Lexical Diversity (MTLD) | 0.019 | 0.053 | 0.022 |
Table 14: Spearman correlations between whole-response embedding similarity and each response-similarity metric on Infinity-Chat , using Qwen3-Embedding-8B, jina-embeddings-v5-text, and QZhou-Embedding ( response pairs). For each metric, pairs that do not meet its computation requirements are excluded. , , . † InfoMatch is computed with the same embedding model as the corresponding column.