Unmasking the Factual-Conceptual Gap in Persian Language Models

Alireza Sakhaeirad; Ali Ma'manpoosh; Arshia Hemmat

cs.CLFeb 19, 2026

Unmasking the Factual-Conceptual Gap in Persian Language Models

Authors: Alireza Sakhaeirad, Ali Ma'manpoosh, Arshia Hemmat

Organizations: EPFL · University of Oxford

Abstract

While emerging Persian NLP benchmarks have expanded into pragmatics and politeness, they rarely distinguish between memorized cultural facts and the ability to reason about implicit social norms. We introduce DivanBench, a diagnostic benchmark focused on superstitions and customs, arbitrary, context-dependent rules that resist simple logical deduction. Through 315 questions across three task types (factual retrieval, paired scenario verification, and situational reasoning), we evaluate seven Persian LLMs and reveal three critical failures: most models exhibit severe acquiescence bias, correctly identifying appropriate behaviors but failing to reject clear violations; continuous Persian pretraining amplifies this bias rather than improving reasoning, often degrading the model's ability to discern contradictions; and all models show a 21% performance gap between retrieving factual knowledge and applying it in scenarios. These findings demonstrate that cultural competence requires more than scaling monolingual data, as current models learn to mimic cultural patterns without internalizing the underlying schemas.

Figures & tables

Category	Example Concepts
Social Etiquette	Taarof , Jang-e Hesab , Doorway Deference, Three-Times Rule, Pishkesh , Shirini
Nowruz Traditions	Haft Sin , Chaharshanbe Suri , Haji Firuz , Samanu Pazan , Khaneh Tekani , Eidi
Supernatural Beings	Jinn , Div , Pari , Bakhtak , Hamzad , Al
Apotropaic Rituals	Nazar amulet, Esfand burning, Salt circle, Bismillah invocations
Wedding Ceremonies	Sofreh Aghd , Knife Dance, Kuzeh Shekani , Honey ritual
Taboos	Whistling at night, Sweeping at night, Stepping on bread, Shoe taboos, Nail cutting restrictions

Table 1: Distribution of cultural concepts across categories. Concepts span ancient Zoroastrian beliefs, Islamic traditions, social etiquette systems, folk cosmology, life-cycle ceremonies, taboos, and divination practices.

Figure 1: Sample Persian cultural concepts from the benchmark, spanning superstitions, traditions, and taboos.

Model	Factual MCQ	Scenario MCQ	Gap	Acquiescence bias
	(Knowledge)	(Reasoning)	(Fact - Scenario)	(Accept - Reject)
Gemma3-12B	87.0 $\pm$ 1.0	66.0 $\pm$ 1.0	21.0	+61.0
Gemma2-9B	83.0 $\pm$ 0.0	65.0 $\pm$ 3.0	18.0	+41.0
Llama3.1-8B	80.0 $\pm$ 2.0	53.0 $\pm$ 5.0	27.0	-10.0
Qwen2.5-7B	75.0 $\pm$ 1.0	60.0 $\pm$ 2.0	15.0	+26.0
Qwen2-7B	73.0 $\pm$ 3.0	48.0 $\pm$ 5.0	25.0	+65.0

Table 2: Core performance metrics across all models. Gap measures difficulty transferring factual knowledge to scenario reasoning. Acquiescence bias (Accept - Reject) reveals pattern-matching: positive values (red) indicate models accept appropriate behavior readily but fail to reject violations. Only Llama3.1-8B shows skeptical bias (blue). Percentages shown; standard deviations computed across 5 prompt variations. Models sorted by Factual MCQ performance.

Figure 2: acquiescence bias across models. Most models accept true cultural statements readily (Blue) but fail to reject false ones (Red), indicating pattern-matching rather than reasoning.

Task	Llama3.1-8B	Dorna2-8B	Change
Reject False	0.73	0.30	-43%
Accept True	0.63	0.91	+28%
Factual MCQ	0.80	0.73	-7%
Scenario MCQ	0.53	0.54	+1%

Table 3: Direct comparison of Llama3.1-8B (base) and Dorna2-8B (Persian-tuned). Persian pretraining dramatically increased acquiescence while destroying critical reasoning.

Figure 3: Factual Knowledge vs. Cultural Reasoning Gap. Gemma3-12B (largest bubble, top-right) achieves highest factual accuracy but shows inconsistent scenario reasoning. All models fall below the diagonal, indicating systematic difficulty in transferring factual knowledge to cultural schema application. Bubble size represents model parameters.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Example Concepts
Social Etiquette	Taarof (ritual politeness), Jang-e Hesab (payment battles), Doorway Deference (older enters first), Three-Times Rule (refuse twice), Pishkesh (gift-giving protocol), Shirini (sweet bringing obligation), Guest-host dynamics, Shoe removal customs, Greeting hierarchies
Superstitions & Omens	Fal-e Hafez (Hafez divination), Fal-Gush (eavesdropping for omens), Dream interpretation, Itchy palms (money coming), Ear ringing (someone talking about you), Twitching eye (bad omen), Sneezing omens, Bird flight patterns, Broken mirrors, Spilled salt
Apotropaic Rituals	Esfand burning (wild rue fumigation), Nazar amulet (evil eye protection), Salt circle protection, Bismillah invocations, Garlic hanging, Iron deterrence, Seven knots ritual, Mirror placement, Knife under pillow, Holy verses
Supernatural Beings	Jinn (fire spirits), Div (demons), Pari (fairies), Bakhtak (sleep paralysis demon), Hamzad (personal spirit double), Al (child-stealing demon), Ghoul , Evil eye personification, Ancestor spirits
Nowruz Traditions	Haft Sin (seven S’s table), Chaharshanbe Suri (fire-jumping), Haji Firuz (blackface herald), Samanu Pazan (wheat pudding stirring), Khaneh Tekani (spring cleaning), Eidi (new year gift), Sizdah Bedar (13th day picnic), Sabzeh growing, Goldfish symbolism, New clothes tradition, Elder visitation order, Fire-jumping prayers, Senjed (jujube significance), Mirror watching, Egg decoration
Wedding Ceremonies	Sofreh Aghd (marriage spread), Honey ritual, Knife dance, Kuzeh Shekani (pot breaking), Sugar cone rubbing, Mirror & candelabra, Needle-sewing ritual, Aghd contract, Shirini Khoran (dessert ceremony), Witness requirements, Gold coin showering, Henna night

Appendix

Table 4: Comprehensive cultural concept coverage. Each concept has paired positive/negative binary scenarios plus representation in factual or scenario MCQs.

Model	Accept True (Positive)	Reject False (Negative)	Bias
Qwen2-7B	84.0 $\pm$ 4.0	19.0 $\pm$ 1.0	+65.0
Gemma3-12B	92.0 $\pm$ 3.0	31.0 $\pm$ 3.0	+61.0
Dorna2-8B	91.0 $\pm$ 5.0	30.0 $\pm$ 5.0	+61.0
Gemma2-9B	84.0 $\pm$ 2.0	43.0 $\pm$ 7.0	+41.0
Qwen2.5-7B	74.0 $\pm$ 7.0	48.0 $\pm$ 6.0	+26.0
Aya-8B	57.0 $\pm$ 4.0	38.0 $\pm$ 5.0	+19.0

Appendix

Table 5: Binary verification task performance sorted by bias magnitude. Models show severe asymmetry: high acceptance of appropriate behavior but low rejection of violations, indicating surface-level pattern-matching.

Explore similar work

TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models
Feb 26, 2026Reihaneh Iranmanesh, Saeedeh Davoudi, Pasha Abrishamchian +2Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
IR Lab, Computer Science Department Georgetown University, Washington D.C.
MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs
Jan 29, 2026Ghazal Kalhor, Behnam BahrakCross-Cultural Language Model Evaluation Figurative Language Understanding
School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Tehran Institute for Advanced Studies, Khatam University, Tehran, Iran
PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions
May 26, 2026Ruhallah Niazi, Faeze Ghorbanpour, Alexander FraserLLM Evaluation Question Answering
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML)

Feb 26, 2026J/K move · Enter open · S save

TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

Reihaneh Iranmanesh, Saeedeh Davoudi, Pasha Abrishamchian, Ophir Frieder+1

IR Lab, Computer Science Department Georgetown University, Washington D.C.

Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation Automated Evaluation Multilingual Language Model Evaluation Question Answering

Open paper AskPDF arXiv

This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models across three culturally grounded Persian datasets, we demonstrate that our hybrid evaluation improves scoring consistency by +10 compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. Our human evaluation further confirms that the proposed semantic similarity metric achieves higher agreement with human judgments than LLM-based judges. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research.

Abstract

Explore similar work

TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions