Artificial Intelligence for Sentiment Analysis of Persian Poetry

Arash Zargar; Abolfazl Moshiri; Mitra Shafaei; Shabnam Rahimi-Golkhandan; Mohamad Tavakoli-Targhi; Farzad Khalvati

cs.CLMar 11, 2026

Artificial Intelligence for Sentiment Analysis of Persian Poetry

Authors: Arash Zargar, Abolfazl Moshiri, Mitra Shafaei, Shabnam Rahimi-Golkhandan, Mohamad Tavakoli-Targhi, Farzad Khalvati

Organizations: Department of Mechanical & Industrial Engineering; Vector Institute for Artificial Intelligence; University of Toronto; Toronto, Canada · Department of Near & Middle Eastern Civilizations; University of Toronto; Toronto, Canada · Independent Researcher · Department of Medical Imaging and Institute of Medical Science; Department of Mechanical & Industrial Engineering; Department of Computer Science; Vector Institute for Artificial Intelligence; University of Toronto; Toronto, Canada

Abstract

Recent advancements of the Artificial Intelligence (AI) have led to the development of large language models (LLMs) that are capable of understanding, analysing, and creating textual data. These language models open a significant opportunity in analyzing the literature and more specifically poetry. In the present work, we employ multiple Bidirectional encoder representations from transformers (BERT) and Generative Pre-trained Transformer (GPT) based language models to analyze the works of two prominent Persian poets: Jalal al-Din Muhammad Rumi (Rumi) and Parvin E'tesami. The main objective of this research is to investigate the capability of the modern language models in grasping complexities of the Persian poetry and explore potential correlations between the poems' sentiment and their meters. Our findings in this study indicates that GPT4o language model can reliably be used in analysis of Persian poetry. Furthermore, the results of our sentiment analysis revealed that in general, Rumi's poems express happier sentiments compared to Parvin E'tesami's poems. Furthermore, comparing the utilization of poetic meters highlighted Rumi's poems superiority in using meters to express a wider variety of sentiments. These findings are significant as they confirm that LLMs can be effectively applied in conducting computer-based semantic studies, where human interpretations are not required, and thereby significantly reducing potential biases in the analysis.

Figures & tables

Table 1: Mapping of poetic meters.

	BERT Multilingual	Pars-Bert	GPT 4o-mini	GPT 4o
Rumi	1.0	1.0	0.9481	0.9578
Parvin	1.0	1.0	0.9882	0.9316

Table 2: Nominal Fleiss’ Kappa parameter based on the seiment scores of vaious language models.

Figure 1: Average Quadratic Weighted Kappa (QWK) of each aggregation method against the individual human annotators. Mean method shows the best performance.

Panel of Figure 2 — Figure 2: Comparison of average sentiment scores assigned by various LLMs to Rumi’s and Parvin E’tesami’s poems. (a) Average sentiment for Rumi’s poems across LLMs. (b) Average sentiment for Parvin E’tesami’s poems across LLMs.(c) Average sentiment for each meter in Rumi’s poems that contain more than 15 poems across LLMs. (d) Average sentiment for each meter in Parvin E’tesami’s poems that contain more than 15 poems across LLMs. The results highlight significant differences in the average sentiment scores returned by different LLMs. Furthermore, it is observed that there is a consistent trend of higher sentiment scores in Rumi’s work regardless of the model used.

Sentiment evaluator	Human annotator 1	Human annotator 2	Human annotator 3	Human annotator 4	Pars-Bert	Bert Multilingual	GPT 4o-mini	GPT 4o
Quadratic Weighted Kappa score	0.8017	0.8025	0.7969	0.7206	0.0006	0.0467	0.5024	0.6003
Absolute Prediction Accuracy (%)	48	47	61	51	6	7	25	33

Table 3: Quadratic Weighted Kappa (QWK) agreement scores and absolute prediction accuracy for human annotators and sentiment analysis models compared against the Ground Truth (Mean aggregation) labels. GPT 4o model shows the highest degree of alignment with human evaluations.

Panel of Figure 3 — Figure 3: Identification of poetic meters associated with positive sentiment in Rumi’s and Parvin E’tesami’s poems. (a) Meters (containing more than 15 poems) with the highest average sentiment scores in Rumi’s poems. (b) Percentage of Rumi’s poems with happy sentiments in the identified meters (containing more than 15 poems). (c) Meters (containing more than 15 poems) with the highest average sentiment scores in Parvin E’tesami’s poems. (d) Percentage of Parvin E’tesami’s poems with happy sentiments in the identified meters (containing more than 15 poems). (e) Comparison of common meters used by both poets, showing distinction between utilizing poems in the poets’ artworks.

Panel of Figure 4 — Figure 4: Sentiment entropy across poetic meters in Rumi’s and Parvin E’tesami’s poems. (a) Entropy of sentiment distribution in Rumi’s poems. (b) Entropy of sentiment distribution in Parvin E’tesami’s poems. The results highlight Rumi’s masterful use of meters to convey diverse sentiments.

Panel of Figure 5 — Figure 5: (a) Standard deviation of sentiment scores in Rumi’s and Parvin E’tesami’s poems, illustrating the variability of emotional expression. Rumi’s poems show a wider range of deviations from the average sentiment. (b) Sentiment polarization in Rumi’s and Parvin E’tesami’s poems, distinguishing between neutral and polarized (very sad or very happy) poems.

Computational Literary Studies

Computational Literary Studies uses data analysis and language technologies to examine literary style, themes, narratives, influence, and change.

See all 48 papers

PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions
May 26, 2026Ruhallah Niazi, Faeze Ghorbanpour, Alexander FraserLLM Evaluation Question Answering
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML)
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
Jun 29, 2026Shanshan Wang, Derek F. Wong, Jingming Yao +1LLM Evaluation LLM-as-a-Judge
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau
Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs
May 21, 2026Shanshan Wang, Fengying Ye, Hanjia Lyu +6Vision-Language Models AI-Generated Text Detection
Department of Computer and Information Science, University of Macau · University of Rochester · Sichuan University +1

May 26, 2026J/K move · Enter open · S save

PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions

Ruhallah Niazi, Faeze Ghorbanpour, Alexander Fraser

School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML)

LLM Evaluation Question Answering Factual Knowledge in Language Models

Open paper AskPDF arXiv

Despite impressive multilingual capabilities, large language models (LLMs) remain poorly evaluated on literary knowledge in non-English languages. We introduce PersLitEval, a benchmark of 4,514 Persian literature multiple-choice questions across eight fine-grained categories spanning spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from materials for the Konkur university entrance examination. We evaluate six LLMs across ten prompting strategies, revealing striking category-level disparities across three tiers of task difficulty: models reach higher accuracy on conceptual similarity tasks but struggle with formal linguistic analysis, with spelling and word formation proving the hardest across all models. Prompting strategy has a significant impact on performance, with explained few-shot examples yielding the best results, particularly on formal linguistic categories. An error analysis identifies three failure modes: semantic comprehension gaps, formal linguistic knowledge gaps, and counting/enumeration errors, suggesting that different categories require different improvement strategies.

Abstract

Computational Literary Studies

Explore similar work

PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions

Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs