MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

Ghazal Kalhor; Behnam Bahrak

cs.CLJan 29, 2026

MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

Authors: Ghazal Kalhor, Behnam Bahrak

Organizations: School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Tehran Institute for Advanced Studies, Khatam University, Tehran, Iran

Abstract

In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the rules of conversational language in order to communicate effectively with users. While previous work has evaluated LLMs' understanding of figurative language in high-resource languages, their performance in low-resource languages remains underexplored. In this paper, we introduce MasalBench, a comprehensive benchmark for assessing LLMs' contextual and cross-cultural understanding of Persian proverbs, which are a key component of conversation in this low-resource language. We evaluate eight state-of-the-art LLMs on MasalBench and find that they perform well in identifying Persian proverbs in context, achieving accuracies above 0.90. However, their performance drops considerably when tasked with identifying equivalent English proverbs, with the best model achieving 0.79 accuracy. Our findings highlight the limitations of current LLMs in cultural knowledge and analogical reasoning, and they provide a framework for assessing cross-cultural understanding in other low-resource languages. MasalBench is available at https://github.com/kalhorghazal/MasalBench.

Figures & tables

Figure 1: Pipeline illustrating the construction of the benchmark for assessing LLMs’ contextual and cross-cultural understanding of Persian proverbs.

Model	Accuracy
Llama 4 Scout	0.906
Llama 3.3 70B Instruct	0.927
Qwen 2.5 72B Instruct	0.911
Qwen QwQ 32B	0.903
DeepSeek V3.1	0.943
DeepSeek R1	0.927

Table 1: Accuracy values for contextual understanding across LLMs.

Figure 2: Heatmap of the proportion of distractor option selections for contextual understanding across LLMs.

Model	Accuracy
Llama 4 Scout	0.647
Llama 3.3 70B Instruct	0.710
Qwen 2.5 72B Instruct	0.691
Qwen QwQ 32B	0.691
DeepSeek V3.1	0.703
DeepSeek R1	0.793

Table 2: Accuracy values for cross-cultural understanding across LLMs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Example prompt for generating dialogues for Persian proverbs, along with its English translation.

Figure 4: Example prompt for generating distractor options in multiple-choice questions for contextual understanding, along with its English translation.

Figure 5: Example prompt for generating English equivalents of Persian proverbs for cross-cultural understanding, along with its English translation.

Figure 6: Example prompt for evaluating LLMs’ contextual understanding of Persian proverbs, along with its English translation.

Figure 7: Example prompt for evaluating LLMs’ cross-cultural understanding of Persian proverbs, along with its English translation.

Explore similar work

TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models
Feb 26, 2026Reihaneh Iranmanesh, Saeedeh Davoudi, Pasha Abrishamchian +2Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
IR Lab, Computer Science Department Georgetown University, Washington D.C.
Unmasking the Factual-Conceptual Gap in Persian Language Models
Feb 19, 2026Alireza Sakhaeirad, Ali Ma'manpoosh, Arshia HemmatCultural Knowledge in Language Models Pragmatic Reasoning in Language Models
EPFL · University of Oxford
LLM Probe: Evaluating LLMs for Low-Resource Languages
Mar 31, 2026Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam, Wolfgang NejdlMultilingual Language Model Evaluation Low-Resource Language Processing
L3S Research Center, Leibniz University Hannover, Germany · Aksum University, Ethiopia

Feb 26, 2026J/K move · Enter open · S save

TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

Reihaneh Iranmanesh, Saeedeh Davoudi, Pasha Abrishamchian, Ophir Frieder+1

IR Lab, Computer Science Department Georgetown University, Washington D.C.

Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation Automated Evaluation Multilingual Language Model Evaluation Question Answering

Open paper AskPDF arXiv

This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models across three culturally grounded Persian datasets, we demonstrate that our hybrid evaluation improves scoring consistency by +10 compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. Our human evaluation further confirms that the proposed semantic similarity metric achieves higher agreement with human judgments than LLM-based judges. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research.

Abstract

Explore similar work

TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

Unmasking the Factual-Conceptual Gap in Persian Language Models

LLM Probe: Evaluating LLMs for Low-Resource Languages