cs.CLApr 4, 2026

POEMetric: The Last Stanza of Humanity

Authors: Bingru Li, Han Wang, Hazel Wilkinson

Organizations: Department of Linguistics and Communication University of Birmingham · Department of Information Engineering and Computer Science University of Trento · Department of English Literature Institute of Data and AI (IDAI) University of Birmingham

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.

Figures & tables

Figure 1: POEMetric. It comprises 10 metrics, including 1) basic instruction-following abilities (form accuracy and theme alignment), 2) advanced creative abilities (creativity, lexical diversity, idiosyncrasy, emotional resonance, and use of literary devices and imagery), and 3) general appraisal (overall poem quality and authorship estimation). Human- and LLM-authored poems are compared through rule-based evaluation and LLM-as-a-judge, whose results are validated by human experts.
Source ×\times FormBalladGhazalLimerickPantoumSestinaSonnetVillanelle
Poetry Foundation8646/66110
Academy of American Poets95/31102
Table 1: The distribution of human poems by form and source
Figure 2: An example of the human poem data and the generation prompt for LLMs. On the left are the related data annotated about a poem, including author, title, poem content, source, form, meter pattern, rhyme pattern, theme, and imagery. Based on these data, the prompt for LLMs to generate poems is curated in the template on the right.
Figure 3: A showcase of the poems by DeepSeek-R1 (Poem A) and a human poet (Poem B) in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.
Figure 4: Chain-of-Thought (CoT) process from DeepSeek-R1 for the poem generation. The model explicitly breaks down the prompt, plans the thematic progression stanza by stanza, brainstorms mter and rhymes, and attempts to strategically insert literary devices.
Figure 5: Rule-based evaluation results. LLMs were able to achieve high form accuracy and MATTR. However, their poems were highly repetitive compared with the original human poems.
Figure 6: Form accuracy and theme alignment scores. Gemini-2.5-Pro achieved the highest scores in both dimensions, whereas Llama-3.3-70B-Instruct ranked the lowest among the 7 LLMs.
Figure 7: Advanced creative abilities. Compared with LLMs, human poets excelled in creativity, idiosyncrasy, emotional resonance, and use of imagery and literary devices.
Figure 8: Overall poem quality and human authorship estimation scores. Humans ranked first in terms of overall poem quality, and human poems remained largely distinguishable from LLM poems.
Figure 9: The mean scores of POEMetric of human poets and 30 LLMs, evaluated by Gemini-2.5-Pro. Models with more parameters within the same family generally performed better. Thinking models were not necessarily better than their non-thinking family members, as GPT-4o and GPT-4 ranked higher than o1 and o3-mini. DeepSeek-R1-Distill models were generally worse than the original models, except Distill-Llama-3.3-70B performed better than its original.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
JudgeHumanDeepSeek-R1Gemini-2.5-ProGPT-4o
Mean2.434.263.003.69
Standard Deviation1.090.200.630.22
Appendix
Table 2: Human vs LLM-as-a-judge evaluation results on the "overall poem quality" dimension.
 Non-ThinkingThinking
Open-sourceDeepSeek-v3 ( Liu et al., 2024 )DeepSeek-R1 ( Guo et al., 2025 )
DeepSeek-R1-Distill-Llama-8B/70B ( Guo et al., 2025 )QwQ-32B ( Team, 2025e )
DeepSeek-R1-Distill-Qwen-1.5B/7B/14B/32B ( Guo et al., 2025 ) 
Gemma-3-27B ( Team et al., 2025 ) 
Llama-3.1-8B/3.3-70B-Instruct ( Grattafiori et al., 2024 ) 
Mistral-Large-2411-123B ( Team, 2024 ) 
Appendix
Table 3: The features of 30 selected LLMs.
Figure 10: The top 20 words across the human and LLM poem datasets.
Figure 11: The top opening words and top imagery cross the human and LLM poem datasets..
Figure 12: A showcase of the poems by Claude-3.7-Sonnet and a human poet in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.
Figure 13: A showcase of the poems by Gemini-2.5-Pro and a human poet in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.
Figure 14: Basic Instruction-Following Abilities, Average Scores
Figure 15: Advanced Creative Abilities, Average Scores
Figure 16: Overall Poem Quality, Average Scores
Figure 17: Human Authorship Estimation, Average Scores
More in this topic

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 382 papers

Explore similar work

CardsList
Aug 29, 2026

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.

Keunhyeung Park, Seunguk Yu, YoungBin Kim

Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University

AI-Generated Text DetectionText GenerationLanguage Model Generation EvaluationRobustness of Machine-Generated Text DetectionStylometry
AskPDF
Oct 16, 2026

Language Models are Lazy Poets: Creative Mode Collapse at Scale

While aligned large language models (LLMs) can successfully complete many creative tasks, their outputs tend to be highly self-similar. Though solutions to this problem of "mode collapse" have been proposed, comprehensive large-scale surveys of the phenomenon in specific textual genres are still lacking. We address this gap through an extensive treatment of LLM mode collapse in poetry. Across 15 poetic forms, 248 subject matter tags, and 16 open-weight models, we generate over 1.1 million poems. To our knowledge, this constitutes the largest corpus of LLM-generated poetry to date. At million-poem scale, we find that LLM poetic vocabularies and rhyming abilities are substantial, but severely underutilized. In other words, LLMs are "lazy" poets: despite possessing large poetic inventories, LLMs over-rely on a small number of words, opening phrases, and rhymes. Further, we find that lexical homogenization differs across models and poetic forms, noting observed rhyming vocabulary is correlated with model size.

Jacob A. Matthews, Sil Hamilton

Cornell University

Output Diversity in Language ModelsMode CollapseText GenerationLLM Evaluation
AskPDF
Jun 29, 2026

Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

Traditional automatic evaluation methods have been shown to be unsuitable for modern Chinese poetry because of the distinct nature of this literary genre. Human evaluation remains reliable, but is expensive and not applicable to large-scale data. In this paper, we propose Poller (Poetry LLM Evaluator), a novel method leveraging large language models (LLMs) to evaluate the poetry understanding task. Specifically, our method requires LLMs to play the role of a poem's author with detailed information, thereby emulating human evaluation and judgment by adopting the poet's perspective. We conducted comprehensive experiments on multiple LLMs, evaluating the interpretations of poems across eight specialized dimensions. Experimental results demonstrate that our method effectively reduces the evaluation error between LLMs and humans. Especially for specific dimension evaluation, Poller-based LLMs achieve a 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization, respectively, compared to baseline methods. These performances are unattainable by conventional LLM evaluation methods. Experimental results from multiple LLMs across various dimensions validate the efficacy of our method. This work bridges the gap between automated efficiency and human expertise, establishing a foundation for automated evaluation in poetry-related tasks.

Shanshan Wang, Derek F. Wong, Jingming Yao +1

NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau

LLM EvaluationLLM-as-a-Judge
AskPDF