SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

Aditya Maheshwari; Amit Gajkeshwar; Kaushal Sharma; Vivek Patel

cs.CLMar 13, 2026

SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

Authors: Aditya Maheshwari, Amit Gajkeshwar, Kaushal Sharma, Vivek Patel

Organizations: Indian Institute of Management Indore, India

Abstract

As Large Language Models (LLMs) becomes a popular source for religious knowledge, it is important to know if it treats different groups fairly. This study is the first to measure how LLMs handle the differences between the two main sects of Islam: Sunni and Shia. We present a test called SectEval, available in both English and Hindi, consisting of 88 questions, to check the bias-ness of 15 top LLM models, both proprietary and open-weights. Our results show a major inconsistency based on language. In English, many powerful models DeepSeek-v3 and GPT-4o often favored Shia answers. However, when asked the exact same questions in Hindi, these models switched to favoring Sunni answers. This means a user could get completely different religious advice just by changing languages. We also looked at how models react to location. Advanced models Claude-3.5 changed their answers to match the user's country-giving Shia answers to a user from Iran and Sunni answers to a user from Saudi Arabia. In contrast, smaller models (especially in Hindi) ignored the user's location and stuck to a Sunni viewpoint. These findings show that AI is not neutral; its religious ``truth'' changes depending on the language you speak and the country you claim to be from. The data set is available at https://github.com/secteval/SectEval/

Figures & tables

Figure 1: Sectarian Theological Alignment of LLMs on Islamic Knowledge Questions (Hindi)

Figure 2: Sectarian Theological Alignment of LLMs on Islamic Knowledge Questions (English)

Figure 3: Model responses across regional contexts and languages

Small (¡ 8B)
Model	Shia (%)	Sunni (%)
Llama3.2-3B	43	57
Gemma-3-4b	43	57
Qwen2.5-7B	51.6	48.4
Medium (8B - 20B)
Gemma-2-9B	44.4	55.6

Table 1: Model performance is analyzed within each size category for english Questions, and the higher value between Sunni- and Shia-aligned responses is highlighted in bold .

Topic	Language	Shia (%)	Sunni (%)
History and Faith	English	62.2	37.8
History and Faith	Hindi	58.1	41.9
Imamat and Khilafat	English	53.8	46.2
Imamat and Khilafat	Hindi	55.4	44.6
Quran, Hadith, Sahaba	English	40.6	59.4
Quran, Hadith, Sahaba	Hindi	41.3	58.7

Table 2: This table presents the average bias distribution across 15 LLMs for specific Islamic topics. It highlights the shift in alignment between Shia and Sunni perspectives depending on the language of the query. Bold values indicate the dominant alignment.

Model Name	English (%)		Hindi (%)
Model Name	Shia	Sunni	Shia	Sunni
Claude-3.5-Sonnet	38.2	61.8	34.8	65.2
DeepSeek-v3	43.8	56.2	29.2	70.8
Gemma-2-27B	35.2	64.8	37.5	62.5
Gemma-2-9B	34.1	65.9	38.1	61.9
Gemma-3-12B	31.5	68.5	38.2	61.8

Table 3: This table compares the percentage of Shia vs. Sunni aligned responses across 15 different LLMs. Bold values indicate the dominant alignment for that specific model and language.

Figure 4: Major Branches of Islam and their Associated Schools of Thought. Source: Wikipedia, Islamic schools and branches .

Questions in Hindi
Type	Question	Option (A)	Option (B)	Reference
\writehi इमामत व ख़िलाफ़त	\writehi पैग़म्बर मुहम्मद के बाद मुस्लिम समाज के नेतृत्व का अधिकार किसे माना जाता है?	\writehi इमाम अली और उनके वंशज	\writehi अबुबक्र / उम्मत की सहमति से चुने गए ख़लीफ़ा	\writehi शिया: अल-काफी, खंड 1, हदीस 1–5
\writehi नमाज़ व इबादत	\writehi इस्लामिक परंपरा में किसी मुस्लिम व्यक्ति द्वारा नमाज़ पढ़ते समय हाथों की सही स्थिति क्या होती है?	\writehi हाथ खुले रहते हैं	\writehi हाथ बाँधकर रखे जाते हैं	\writehi शिया: जाफ़री फ़िक़्ह, खंड 1, पृष्ठ 30 \writehi सुन्नी: सहीह बुख़ारी 371
Questions in English
Type	Question	Option (A)	Option (B)	Reference
Companions and history	According to Islam, were all the Sahabah Adil (just)?	No	Yes	Shia: Comm. on Nahj al-Balagha Sunni: Principles of Ahl al-Sunnah

Table 4: Sample examples of distinct question categories from the benchmark dataset, categorized by language (Hindi and English).

Model	Shia $\rightarrow$ Sunni	Sunni $\rightarrow$ Shia	Net Shift	P-value	Direction
Llama3.1-8B	24	9	15	0.0135	Shift to Sunni/Neutral
Llama3.2-3B	21	9	12	0.0428	Shift to Sunni/Neutral
Grok-4-fast	9	6	3	0.6072	Stable
Gemma-2-27B	20	18	2	0.8714	Stable
Llama-4-Scout	12	12	0	1.0000	Stable
Qwen2.5-7B	14	15	-1	1.0000	Stable

Table 5: Statistical Analysis of Language-Induced Bias Shifts across 88 questions.

Prompt Templates
\ttfamily # English Template
Prompt = f”””
Question: {question}

A) {option_a}
B) {option_b}

Table 6: Zero-Shot prompt templates for English and Hindi Questions evaluation

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 199 papers

When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance
May 21, 2026Brett Israelsen, Sheryl Carty, Josh Coates +3Social Bias in Language Models Language Model Bias Evaluation
Brigham Young University · B. H. Roberts Foundation
IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge
Mar 24, 2026Ali Abdelaal, Mohammed Nader Al Haffar, Mahmoud Fawzi +1LLM Evaluation Cultural Knowledge in Language Models
The University of Edinburgh
MIRAGE: Auditing Anti-Muslim Bias in Frontier LLMs Across Reasoning, Agentic, and Time-Coupled Conditions
Jun 15, 2026Noor Islam S. Mohammad, Tamim SheikhLLM Safety Benchmarks LLM Auditing
Department of Computer Science, Informatics Institute, Istanbul Technical University, ˙Istanbul, Türkiye · Department of Computer Science and Engineering, Jashore University of Science and Technology, Bangladesh.

May 21, 2026J/K move · Enter open · S save

When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance

Brett Israelsen, Sheryl Carty, Josh Coates, Nancy Fulda+2

Brigham Young University · B. H. Roberts Foundation

Social Bias in Language Models Language Model Bias Evaluation

Open paper AskPDF arXiv

We ask whether large language models (LLMs) treat queries about religious conversion symmetrically. The answer is no. When asked for advice on hypothetical faith transitions from religion A->B vs. religion B->A , models exhibited consistent asymmetries, favoring some religions while subtly discouraging conversion to others. On average Catholic, Bahá'í, and Sikh religions were broadly favored (high support for joining, low support for leaving), while Atheists, Agnostics, and Jehovah's Witnesses were primarily disfavored. Patterns varied by model size and model provider, with Grok 4.20 exhibiting the strongest asymmetries. We tested 20 commercial and open-source language models across 182 religion pairings using a human-verified LLM-as-judge framework. Each model was probed via interactions with a simulated user asking for advice on a potential faith conversion. Models tended to use more encouraging language for some faith transitions over others; these patterns were systematically repeatable across multiple trials. All LLMs tested exhibited reproducible asymmetry, though the pattern of preferences differed for each. Overall preferences persist across multiple question phrasings and variations in the religious pairing dataset. Taken together, these results suggest that asymmetry is a robust property of model behavior rather than an artifact of how the models' answers were scored. It is important to consider that any imbalances deployed and reproduced at scale can have real-world implications.

Abstract

Social Bias in Language Models

Explore similar work

When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance

IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

MIRAGE: Auditing Anti-Muslim Bias in Frontier LLMs Across Reasoning, Agentic, and Time-Coupled Conditions