POEMetric: The Last Stanza of Humanity

Bingru Li; Han Wang; Hazel Wilkinson

cs.CLApr 4, 2026

POEMetric: The Last Stanza of Humanity

Authors: Bingru Li, Han Wang, Hazel Wilkinson

Organizations: Department of Linguistics and Communication University of Birmingham · Department of Information Engineering and Computer Science University of Trento · Department of English Literature Institute of Data and AI (IDAI) University of Birmingham

Abstract

Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.

Figures & tables

Figure 1: POEMetric. It comprises 10 metrics, including 1) basic instruction-following abilities (form accuracy and theme alignment), 2) advanced creative abilities (creativity, lexical diversity, idiosyncrasy, emotional resonance, and use of literary devices and imagery), and 3) general appraisal (overall poem quality and authorship estimation). Human- and LLM-authored poems are compared through rule-based evaluation and LLM-as-a-judge, whose results are validated by human experts.

Source $\times$ Form	Ballad	Ghazal	Limerick	Pantoum	Sestina	Sonnet	Villanelle
Poetry Foundation	86	4	6	/	6	61	10
Academy of American Poets	9	5	/	3	1	10	2

Table 1: The distribution of human poems by form and source

Figure 2: An example of the human poem data and the generation prompt for LLMs. On the left are the related data annotated about a poem, including author, title, poem content, source, form, meter pattern, rhyme pattern, theme, and imagery. Based on these data, the prompt for LLMs to generate poems is curated in the template on the right.

Figure 3: A showcase of the poems by DeepSeek-R1 (Poem A) and a human poet (Poem B) in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.

Figure 4: Chain-of-Thought (CoT) process from DeepSeek-R1 for the poem generation. The model explicitly breaks down the prompt, plans the thematic progression stanza by stanza, brainstorms mter and rhymes, and attempts to strategically insert literary devices.

Figure 5: Rule-based evaluation results. LLMs were able to achieve high form accuracy and MATTR. However, their poems were highly repetitive compared with the original human poems.

Figure 6: Form accuracy and theme alignment scores. Gemini-2.5-Pro achieved the highest scores in both dimensions, whereas Llama-3.3-70B-Instruct ranked the lowest among the 7 LLMs.

Panel of Figure 7 — Figure 7: Advanced creative abilities. Compared with LLMs, human poets excelled in creativity, idiosyncrasy, emotional resonance, and use of imagery and literary devices.

Figure 8: Overall poem quality and human authorship estimation scores. Humans ranked first in terms of overall poem quality, and human poems remained largely distinguishable from LLM poems.

Figure 9: The mean scores of POEMetric of human poets and 30 LLMs, evaluated by Gemini-2.5-Pro. Models with more parameters within the same family generally performed better. Thinking models were not necessarily better than their non-thinking family members, as GPT-4o and GPT-4 ranked higher than o1 and o3-mini. DeepSeek-R1-Distill models were generally worse than the original models, except Distill-Llama-3.3-70B performed better than its original.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Judge	Human	DeepSeek-R1	Gemini-2.5-Pro	GPT-4o
Mean	2.43	4.26	3.00	3.69
Standard Deviation	1.09	0.20	0.63	0.22

Appendix

Table 2: Human vs LLM-as-a-judge evaluation results on the "overall poem quality" dimension.

	Non-Thinking	Thinking
Open-source	DeepSeek-v3 ( Liu et al., 2024 )	DeepSeek-R1 ( Guo et al., 2025 )
	DeepSeek-R1-Distill-Llama-8B/70B ( Guo et al., 2025 )	QwQ-32B ( Team, 2025e )
	DeepSeek-R1-Distill-Qwen-1.5B/7B/14B/32B ( Guo et al., 2025 )
	Gemma-3-27B ( Team et al., 2025 )
	Llama-3.1-8B/3.3-70B-Instruct ( Grattafiori et al., 2024 )
	Mistral-Large-2411-123B ( Team, 2024 )

Appendix

Table 3: The features of 30 selected LLMs.

Figure 10: The top 20 words across the human and LLM poem datasets.

Figure 11: The top opening words and top imagery cross the human and LLM poem datasets..

Figure 12: A showcase of the poems by Claude-3.7-Sonnet and a human poet in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.

Figure 13: A showcase of the poems by Gemini-2.5-Pro and a human poet in response to the same prompt. The bar charts show their POEMetric scores judged by Gemini-2.5-Pro.

Figure 14: Basic Instruction-Following Abilities, Average Scores

Figure 15: Advanced Creative Abilities, Average Scores

Figure 16: Overall Poem Quality, Average Scores

Figure 17: Human Authorship Estimation, Average Scores

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 382 papers

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features
Aug 29, 2026Keunhyeung Park, Seunguk Yu, YoungBin KimAI-Generated Text Detection Text Generation
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University
Language Models are Lazy Poets: Creative Mode Collapse at Scale
Oct 16, 2026Jacob A. Matthews, Sil HamiltonOutput Diversity in Language Models Mode Collapse
Cornell University
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
Jun 29, 2026Shanshan Wang, Derek F. Wong, Jingming Yao +1LLM Evaluation LLM-as-a-Judge
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau

Aug 29, 2026J/K move · Enter open · S save

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

Keunhyeung Park, Seunguk Yu, YoungBin Kim

Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University

AI-Generated Text Detection Text Generation Language Model Generation Evaluation Robustness of Machine-Generated Text Detection Stylometry

Open paper AskPDF arXiv

LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.

Abstract

Language Model Generation Evaluation

Explore similar work

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

Language Models are Lazy Poets: Creative Mode Collapse at Scale

Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?