Can ChatGPT Really Understand Modern Chinese Poetry?

Shanshan Wang; Derek F. Wong; Jingming Yao; Lidia S. Chao

cs.CLMar 21, 2026

Can ChatGPT Really Understand Modern Chinese Poetry?

Authors: Shanshan Wang, Derek F. Wong, Jingming Yao, Lidia S. Chao

Organizations: NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau

Abstract

ChatGPT has demonstrated remarkable capabilities on both poetry generation and translation, yet its ability to truly understand poetry remains unexplored. Previous poetry-related work merely analyzed experimental outcomes without addressing fundamental issues of comprehension. This paper introduces a comprehensive framework for evaluating ChatGPT's understanding of modern poetry. We collaborated with professional poets to evaluate ChatGPT's interpretation of modern Chinese poems by different poets along multiple dimensions. Evaluation results show that ChatGPT's interpretations align with the original poets' intents in over 73% of the cases. However, its understanding in certain dimensions, particularly in capturing poeticity, proved to be less satisfactory. These findings highlight the effectiveness and necessity of our proposed framework. This study not only evaluates ChatGPT's ability to understand modern poetry but also establishes a solid foundation for future research on LLMs and their application to poetry-related tasks.

Figures & tables

Figure 1: The framework for evaluating ChatGPT’s understanding of modern poetry.

Please understand the following modern poem titled

T_{i}

from the five dimensions of content, expression methods, thought and emotion, modernity, and the most poetic sentence, and finally give your interpretation of the poem. In addition to content, please provide examples and specific explanations for each dimension. Please note that this is a complete modern poem. The specific definitions of each dimension are as follows:

Content: Summarize what the poem is about.

Expression Methods: Understand how this poem is written from five aspects: language, imagery, rhetorical techniques, rhythm, and defamiliarization. The five specific aspects are as follows: 1) What are the characteristics of the language of this poem? Is it innovative? 2) What imageries are used in this poem and what are their functions? 3) What rhetorical techniques is used in this poem? What is the function? 4) How is the rhythm of this poem? Such as pauses, lines, and stanzas. 5) Does this poem contain any expression of defamiliarization? If there is any unfamiliar expression, please point it out; if not, you can skip it.

Thought & Emotion: What thought and emotion does this poem express?

Modernity: Is the poem modern or modern-minded?

Poeticity: Choose the most poetic sentence in this poem.

Poem: {{

P_{i}

}} (Directly used the original Chinese poem)

Table 1: The prompt we designed for ChatGPT to understand modern poetry.

	Cont	Lang	Imag	Rhet	Rhyt	Defa	Thou	Mode
Com-Poems	80.33	79.05	81.18	77.83	76.15	79.40	78.80	79.88
Spe-Poems	77.50	73.75	81.25	88.75	82.50	77.50	78.75	82.50

Table 2: Evaluation results of different types of poems by the original poets.

	N-100	N-0	N-50
Com-Poems	17	2	21
Spe-Poems	3	1	4

Table 3: Evaluation results of ChatGPT’s understanding of poeticity. “N-100”, “N-0” and “N-50” respectively represent the number of occurrences of 100, 0, and 50.

Figure 2: The distribution of evaluation scores for GPT-4’s interpretation of 48 poems.

Please score the following “readers’ understanding of the poem” from the eight dimensions of content, language, imagery, rhetorical techniques, rhythm, defamiliarization, thought and emotion, and modernity, and give detailed reasons for your score. Please note that you only need to score and give reasons for the “Readers’ Understanding of this poem, not the poem itself.

Scoring Dimensions: Same as human evaluation.

Scoring Rules: Each dimension ranges from 0 to 100. A score of 0 means that you think the reader’s understanding of a certain dimension is completely wrong, and 100 indicates that you think the reader’s understanding of a certain dimension is completely correct; the more correct the reader’s understanding of a certain dimension is, the higher the score you will give.

Poem: {{

P_{i}

}}

Readers’ Understanding of this Poem: {{

U_{i}

}}

Table 4: The prompt we designed for the LLMs as the evaluators to evaluate GPT-4’s understanding of poems.

	Cont	Lang	Imag	Rhet	Rhyt	Defa	Thou	Mode
o1-preview	94.70	92.03	96.33	91.40	90.35	93.90	95.23	96.75
GPT-4o	92.38	89.38	94.00	88.50	88.00	87.38	91.75	93.25
Qwen2.5-Plus	94.50	89.50	94.50	90.00	85.13	91.25	90.25	94.88
Qwen2.5-Max	95.00	89.13	94.13	93.00	90.00	86.13	95.00	95.00

Table 5: The evaluation results of LLMs as the evaluators on GPT-4’s understanding of Com-Poems.

	Cont	Lang	Imag	Rhet	Rhyt	Defa	Thou	Mode
o1-preview	95.00	92.50	96.25	88.75	93.13	83.13	96.25	94.38
GPT-4o	91.88	91.25	94.38	87.50	93.75	81.88	92.50	93.75
Qwen2.5-Plus	95.00	90.00	95.00	90.00	85.00	91.25	90.00	95.00
Qwen2.5-Max	95.00	90.00	95.00	90.00	90.00	85.00	95.00	95.00

Table 6: The evaluation results of LLMs as the evaluators on GPT-4’s understanding of Spe-Poems.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?
Aug 22, 2026Tian Lan, Shanshan Wang, Zehua Duo +4LLM Evaluation Computational Literary Studies
Graduate School of Informatics, Kyoto University · NLP2CT Lab, Department of Computer and Information Science, University of Macau · College of Computer Science, Inner Mongolia University
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
Jun 29, 2026Shanshan Wang, Derek F. Wong, Jingming Yao +1LLM Evaluation LLM-as-a-Judge
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau
Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs
May 21, 2026Shanshan Wang, Fengying Ye, Hanjia Lyu +6Vision-Language Models AI-Generated Text Detection
Department of Computer and Information Science, University of Macau · University of Rochester · Sichuan University +1

Aug 22, 2026J/K move · Enter open · S save

Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?

Tian Lan, Shanshan Wang, Zehua Duo, Jiang Li+3

Graduate School of Informatics, Kyoto University · NLP2CT Lab, Department of Computer and Information Science, University of Macau · College of Computer Science, Inner Mongolia University

LLM Evaluation Computational Literary Studies

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks, yet their ability to understand literary texts, particularly modern Chinese poetry, remains largely unexplored. The unique literary characteristics of modern Chinese poetry necessitate a distinct form of reasoning for effective comprehension. Unlike conventional texts that convey clear information, the unique "poetic logic" of modern Chinese poetry requires a holistic reasoning approach that goes beyond superficial semantic analysis to be understood. However, current evaluation paradigms largely ignore this critical dimension. To address this gap, we propose Peony, the first benchmark specifically designed for evaluating the poetic logic of modern Chinese poetry. We define poetic logic as four tasks across three levels, namely stanza, line, and imagery, and systematically evaluate and analyze six mainstream LLMs based on Peony. We evaluate these models under both non-thinking and thinking configurations. The experimental results reveal the limitations of current LLMs in understanding the poetic logic of modern Chinese poetry and validate the effectiveness and necessity of Peony. Our data and code will be available.

Abstract

LLM Evaluation

Explore similar work

Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?

Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs