Organizations: Department of Mechanical & Industrial Engineering; Vector Institute for Artificial Intelligence; University of Toronto; Toronto, Canada · Department of Near & Middle Eastern Civilizations; University of Toronto; Toronto, Canada · Independent Researcher · Department of Medical Imaging and Institute of Medical Science; Department of Mechanical & Industrial Engineering; Department of Computer Science; Vector Institute for Artificial Intelligence; University of Toronto; Toronto, Canada
Recent advancements of the Artificial Intelligence (AI) have led to the development of large language models (LLMs) that are capable of understanding, analysing, and creating textual data. These language models open a significant opportunity in analyzing the literature and more specifically poetry. In the present work, we employ multiple Bidirectional encoder representations from transformers (BERT) and Generative Pre-trained Transformer (GPT) based language models to analyze the works of two prominent Persian poets: Jalal al-Din Muhammad Rumi (Rumi) and Parvin E'tesami. The main objective of this research is to investigate the capability of the modern language models in grasping complexities of the Persian poetry and explore potential correlations between the poems' sentiment and their meters. Our findings in this study indicates that GPT4o language model can reliably be used in analysis of Persian poetry. Furthermore, the results of our sentiment analysis revealed that in general, Rumi's poems express happier sentiments compared to Parvin E'tesami's poems. Furthermore, comparing the utilization of poetic meters highlighted Rumi's poems superiority in using meters to express a wider variety of sentiments. These findings are significant as they confirm that LLMs can be effectively applied in conducting computer-based semantic studies, where human interpretations are not required, and thereby significantly reducing potential biases in the analysis.
Figures & tables
Table 1: Mapping of poetic meters.
BERT Multilingual
Pars-Bert
GPT 4o-mini
GPT 4o
Rumi
1.0
1.0
0.9481
0.9578
Parvin
1.0
1.0
0.9882
0.9316
Table 2: Nominal Fleiss’ Kappa parameter based on the seiment scores of vaious language models.
Figure 1: Average Quadratic Weighted Kappa (QWK) of each aggregation method against the individual human annotators. Mean method shows the best performance.
Figure 2: Comparison of average sentiment scores assigned by various LLMs to Rumi’s and Parvin E’tesami’s poems. (a) Average sentiment for Rumi’s poems across LLMs. (b) Average sentiment for Parvin E’tesami’s poems across LLMs.(c) Average sentiment for each meter in Rumi’s poems that contain more than 15 poems across LLMs. (d) Average sentiment for each meter in Parvin E’tesami’s poems that contain more than 15 poems across LLMs. The results highlight significant differences in the average sentiment scores returned by different LLMs. Furthermore, it is observed that there is a consistent trend of higher sentiment scores in Rumi’s work regardless of the model used.
Sentiment evaluator
Human annotator 1
Human annotator 2
Human annotator 3
Human annotator 4
Pars-Bert
Bert Multilingual
GPT 4o-mini
GPT 4o
Quadratic Weighted Kappa score
0.8017
0.8025
0.7969
0.7206
0.0006
0.0467
0.5024
0.6003
Absolute Prediction Accuracy (%)
48
47
61
51
6
7
25
33
Table 3: Quadratic Weighted Kappa (QWK) agreement scores and absolute prediction accuracy for human annotators and sentiment analysis models compared against the Ground Truth (Mean aggregation) labels. GPT 4o model shows the highest degree of alignment with human evaluations.
Figure 3: Identification of poetic meters associated with positive sentiment in Rumi’s and Parvin E’tesami’s poems. (a) Meters (containing more than 15 poems) with the highest average sentiment scores in Rumi’s poems. (b) Percentage of Rumi’s poems with happy sentiments in the identified meters (containing more than 15 poems). (c) Meters (containing more than 15 poems) with the highest average sentiment scores in Parvin E’tesami’s poems. (d) Percentage of Parvin E’tesami’s poems with happy sentiments in the identified meters (containing more than 15 poems). (e) Comparison of common meters used by both poets, showing distinction between utilizing poems in the poets’ artworks.
Figure 4: Sentiment entropy across poetic meters in Rumi’s and Parvin E’tesami’s poems. (a) Entropy of sentiment distribution in Rumi’s poems. (b) Entropy of sentiment distribution in Parvin E’tesami’s poems. The results highlight Rumi’s masterful use of meters to convey diverse sentiments.
Figure 5: (a) Standard deviation of sentiment scores in Rumi’s and Parvin E’tesami’s poems, illustrating the variability of emotional expression. Rumi’s poems show a wider range of deviations from the average sentiment. (b) Sentiment polarization in Rumi’s and Parvin E’tesami’s poems, distinguishing between neutral and polarized (very sad or very happy) poems.
Despite impressive multilingual capabilities, large language models (LLMs) remain poorly evaluated on literary knowledge in non-English languages. We introduce PersLitEval, a benchmark of 4,514 Persian literature multiple-choice questions across eight fine-grained categories spanning spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from materials for the Konkur university entrance examination. We evaluate six LLMs across ten prompting strategies, revealing striking category-level disparities across three tiers of task difficulty: models reach higher accuracy on conceptual similarity tasks but struggle with formal linguistic analysis, with spelling and word formation proving the hardest across all models. Prompting strategy has a significant impact on performance, with explained few-shot examples yielding the best results, particularly on formal linguistic categories. An error analysis identifies three failure modes: semantic comprehension gaps, formal linguistic knowledge gaps, and counting/enumeration errors, suggesting that different categories require different improvement strategies.
Ruhallah Niazi, Faeze Ghorbanpour, Alexander Fraser
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML)
Traditional automatic evaluation methods have been shown to be unsuitable for modern Chinese poetry because of the distinct nature of this literary genre. Human evaluation remains reliable, but is expensive and not applicable to large-scale data. In this paper, we propose Poller (Poetry LLM Evaluator), a novel method leveraging large language models (LLMs) to evaluate the poetry understanding task. Specifically, our method requires LLMs to play the role of a poem's author with detailed information, thereby emulating human evaluation and judgment by adopting the poet's perspective. We conducted comprehensive experiments on multiple LLMs, evaluating the interpretations of poems across eight specialized dimensions. Experimental results demonstrate that our method effectively reduces the evaluation error between LLMs and humans. Especially for specific dimension evaluation, Poller-based LLMs achieve a 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization, respectively, compared to baseline methods. These performances are unattainable by conventional LLM evaluation methods. Experimental results from multiple LLMs across various dimensions validate the efficacy of our method. This work bridges the gap between automated efficiency and human expertise, establishing a foundation for automated evaluation in poetry-related tasks.
Shanshan Wang, Derek F. Wong, Jingming Yao +1
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau
Previous detection studies have shown that LLMs cannot be effectively used as detectors, but these studies have not addressed modern Chinese poetry. Moreover, no relevant research has explored the performance of LLMs in detecting modern Chinese poetry. This paper evaluates and enhances the performance of LLMs as detectors for modern Chinese poetry, and proposes an image-semantic guided poetry detection method. Compared with traditional detection approaches, our method innovatively incorporates images that reflect the content of the poetry. Through example-driven approaches, our method effectively integrates information such as meaning, imagery, and feeling from the image, then forms a complementary judgment with the poem text. Experimental results demonstrate that the LLM detectors based on our method outperform baseline detectors based on plain text, and even surpass the best-performing traditional detector, RoBERTa. The Gemini detector using our method achieves a Macro-F1 score of 85.65%, reaching the state-of-the-art level. The performance improvements of different LLM detectors on multiple LLMs-generated data prove the effectiveness of our method.
Shanshan Wang, Fengying Ye, Hanjia Lyu +6
Department of Computer and Information Science, University of Macau · University of Rochester · Sichuan University +1