Language Models are Lazy Poets: Creative Mode Collapse at Scale

Jacob A. Matthews; Sil Hamilton

cs.CLOct 16, 2026

Language Models are Lazy Poets: Creative Mode Collapse at Scale

Authors: Jacob A. Matthews, Sil Hamilton

Organizations: Cornell University

Abstract

While aligned large language models (LLMs) can successfully complete many creative tasks, their outputs tend to be highly self-similar. Though solutions to this problem of "mode collapse" have been proposed, comprehensive large-scale surveys of the phenomenon in specific textual genres are still lacking. We address this gap through an extensive treatment of LLM mode collapse in poetry. Across 15 poetic forms, 248 subject matter tags, and 16 open-weight models, we generate over 1.1 million poems. To our knowledge, this constitutes the largest corpus of LLM-generated poetry to date. At million-poem scale, we find that LLM poetic vocabularies and rhyming abilities are substantial, but severely underutilized. In other words, LLMs are "lazy" poets: despite possessing large poetic inventories, LLMs over-rely on a small number of words, opening phrases, and rhymes. Further, we find that lexical homogenization differs across models and poetic forms, noting observed rhyming vocabulary is correlated with model size.

Figures & tables

Figure 1: To investigate mode collapse at scale we generate ca. 1.1 million poems. We find that while the poems appear largely collapsed when viewed in aggregate, remarkable diversity persists in the long tail.

Model	Year	Size (B)
Qwen3 ( Yang et al., 2025 )	2025	4/8/14/30/32
Llama 3.1 ( Dubey et al., 2024 )	2024	8
Llama 3.2 ( Meta, 2024 )	2024	3
Ministral 3 ( Liu et al., 2026 )	2025	3/8/14
Gemma 3 ( Kamath et al., 2025 )	2025	4/12/27
OLMo 3 ( Ettinger et al., 2025 )	2025	7

Table 1: Model families and sizes considered in our experiment. All models listed generated prompts, while those above the separator also generated poems. In all cases we use the final instruct-tuned variant.

Panel of Figure 2 — Figure 2: An example prompt together with two sample responses from the dataset. Note the opening phrase is repeated almost exactly between responses, even though the generators belong to different model families.

Figure 3: Mean lexical homogenization scores $\pm 1$ SE, averaged by poetic form (top) and generator model (bottom), including baseline scores for randomly sampled form, generator, and 5M random pairs ("Random").

Figure 4: Union of the top-5 most commonly featured content words in generated poems. Percentages indicate share of poems by a generator that contain the listed word. The share of models with the word in their top 5 in brackets.

Figure 5: Percent of poems by a given model family that begin with the reported trigram. We include the union of all models’ most frequent opening trigrams.

	Correlation with model size
	Rhyming words		Rhyming pairs		Rhyming vocabulary share
Threshold	Corr.	$p$ -value	Corr.	$p$ -value	Corr.	$p$ -value
$\leq 2$	0.765	0.001	0.735	0.001	0.471	0.065
$\leq 4$	0.765	0.001	0.737	0.001	0.472	0.065
$\leq 8$	0.763	0.001	0.737	0.001	0.470	0.066
Unrestricted	0.752	0.001	0.719	0.002	0.477	0.061

Table 2: Pearson correlations between model size and rhyme measures at each line distance threshold. Rhyming vocabulary share is the number of distinct rhyming words divided by total observed word types for each model.

Figure 6: Cumulative concentration curves showing the cumulative share of document occurrences accounted for by the most frequent types. Lines closer to the top-left indicate greater collapse. Solid lines show rhyming pairs ( $\leq 2$ line threshold), and dashed lines show content words. As an approximate human baseline, we include pooled results from the Poetry Foundation dataset by Walsh et al. (2024a) (gray). In each cell, we also report the cumulative share of occurrences accounted for by the top 1% of word types ( Walsh et al. Pairs: 11.87%, Words: 24.89%).

Figure 7: Model-unique vocabulary (word types appearing only in poems by a single model) grouped by family. We report the number and total vocabulary share of model-unique types, as well as the unique types appearing in the most poems per model family. Values on the $x$ axis indicate the total number of poems where a given type appears.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Operation	%
Any cleaning	38.65
Title	28.55
Introduction	3.41
Postscript	4.82
Decoration/annotation	23.49
Refusal	0.03

Appendix

Table 3: Percent of generated poems affected by data cleaning operations.

Figure 8: Percent of poems with titles removed during cleaning per model.

Title	Poems	Models	Top Generator (count)
The Weight of Light	1,219	11	Ministral 3 3B (426)
The Weight of What Was	1,042	10	Ministral 3 14B (441)
The Weaver’s Thread	953	14	Ministral 3 14B (265)
Elegy for the Unseen	837	13	Qwen3 30B-A3B (255)
The Weight of Wings	803	14	Ministral 3 8B (321)
The Weight of Hands	714	10	Ministral 3 3B (273)

Appendix

Table 4: Top-10 most-repeated titles generated by any model.

Figure 9: Mean word length of generated prompts by model.

Figure 10: Distinct words present in line-terminal rhyming pairs (left), and total distinct rhyming pairs (right). Color corresponds to the maximum line distance rule for counting pairs.

Figure 11: Most frequent words appearing uniquely in poems generated by a given model. Beneath each model name, we also report the total number of model-unique words observed for that model.

Output Diversity in Language Models

See all 32 papers

POEMetric: The Last Stanza of Humanity
Apr 4, 2026Bingru Li, Han Wang, Hazel WilkinsonLanguage Model Generation Evaluation Generative AI Evaluation
Department of Linguistics and Communication University of Birmingham · Department of Information Engineering and Computer Science University of Trento · Department of English Literature Institute of Data and AI (IDAI) University of Birmingham
Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features
Aug 29, 2026Keunhyeung Park, Seunguk Yu, YoungBin KimAI-Generated Text Detection Text Generation
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University
Interpretable Stylistic Variation in Human and LLM Writing Across Genres, Models, and Decoding Strategies
Apr 15, 2026Swati Rallapalli, Shannon Gallagher, Ronald Yurko +4Stylometry Language Model Generation Evaluation
Software Engineering Institute, AI Division, Carnegie Mellon University · Department of Statistics & Data Science, Carnegie Mellon University

Apr 4, 2026J/K move · Enter open · S save

POEMetric: The Last Stanza of Humanity

Bingru Li, Han Wang, Hazel Wilkinson

Department of Linguistics and Communication University of Birmingham · Department of Information Engineering and Computer Science University of Trento · Department of English Literature Institute of Data and AI (IDAI) University of Birmingham

Language Model Generation Evaluation Generative AI Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.

Abstract

Output Diversity in Language Models

Explore similar work

POEMetric: The Last Stanza of Humanity

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

Interpretable Stylistic Variation in Human and LLM Writing Across Genres, Models, and Decoding Strategies