Large Language Models (LLMs) have made significant progress in reasoning tasks across various domains such as mathematics and coding. However, their performance deteriorates in tasks requiring rich socio-cultural knowledge and diverse local contexts, particularly those involving Indian Culture. Existing Cultural benchmarks are (i) Manually crafted, (ii) contain single-hop questions testing factual recall, and (iii) prohibitively costly to scale, leaving this deficiency largely unmeasured. To address this, we introduce VIRAASAT, a novel, semi-automated multi-hop approach for generating cultural specific multi-hop Question-Answering dataset for Indian culture. VIRAASAT leverages a Knowledge Graph comprising more than 700 expert-curated cultural artifacts, covering 13 key attributes of Indian culture (history, festivals, etc). VIRAASAT spans all 28 states and 8 Union Territories, yielding more than 3,200 multi-hop questions that necessitate chained cultural reasoning. We evaluate current State-of-the-Art (SOTA) LLMs on VIRAASAT and identify key limitations in reasoning wherein fine-tuning on Chain-of-Thought(CoT) traces fails to ground and synthesize low-probability facts. To bridge this gap, we propose a novel framework named Symbolic Chain-of-Manipulation (SCoM). Adapting the Chain-of-Manipulation paradigm, we train the model to simulate atomic Knowledge Graph manipulations internally. SCoM teaches the model to reliably traverse the topological structure of the graph. Experiments on Supervised Fine-Tuning (SFT) demonstrate that SCoM outperforms standard CoT baselines by up to 20%. We release the VIRAASAT dataset along with our findings, laying a strong foundation towards building Culturally Aware Reasoning Models.
Figures & tables
Figure 1. VIRAASAT dataset construction pipeline: artifact curation and identifier creation, Knowledge Graph construction, template-based 2-hop question generation, and expert verification for semantic and grammatical quality.
Figure 2. State-wise and attribute-wise distribution of questions in VIRAASAT .
Question
Graph Path (Anchor → State → Target)
Answer
Chain-of-Thought Trace
Which dish, a savoury fried bread made primarily from powdered rice, originated in the same state where the festival Baha Parab (the spring festival of the Santhal tribe) is celebrated?
(1) Garad saree is from West Bengal. (2) Mishti doi is from West Bengal. (3) Therefore, the dish is Mishti doi.
Which sacred ghat, located on the banks of the Ganges and considered one of the most important pilgrimage sites, is situated in the same state where the “Ramman festival” is celebrated?
(1) Ramman festival is celebrated in Uttarakhand. (2) Har ki Pauri is in Uttarakhand. (3) Therefore, the ghat is Har ki Pauri.
Table 1. Sample data instances from VIRAASAT dataset. Each question requires traversing a constrained 2-hop path from an anchor entity to a bridge state and then to the target artifact.
Dataset
# Train samples
# Val samples
# Test samples
VIRAASAT
2252
323
643
Total
3218 questions
Table 2. Dataset statistics for VIRAASAT .
Figure 3. SCoM reasoning-trace generation pipeline: The figure illustrates the SCoM reasoning trace generation pipeline. The Student Agent (actor) solves the question by producing an explicit, step-wise trace consisting of THOUGHT and ACTION steps, where each ACTION represents an atomic manipulation over the cultural knowledge graph (entity grounding, enforcing the state constraint, candidate retrieval and resolution, and target resolution). Each proposed action is checked by the Symbolic Verifier (teacher) that validates graph-topology consistency with the constrained path; if an action violates constraints, the verifier injects a correction signal that redirects the next step, preventing drift from the intended traversal. Valid actions are executed against the parametric environment backed by the cultural knowledge graph, returning OBSERVATION outputs such as candidate entities and textual descriptors used to guide subsequent steps. The actor–verifier loop iterates for a bounded number of turns (k=5), producing a faithful, tool-grounded SCoM reasoning trace that is used for downstream SFT, teaching a model to perform structured graph-grounded retrieval and reasoning.
Figure 4. Example SCoM trace (cleaned) illustrating tool-grounded retrieval over a constrained 2-hop path.
Figure 5. Prompt used for SFT, providing the necessary task context and details.
Model
State Match (%)
Answer Match (%)
Full Match (%)
Zero-shot
gemini-2.0-flash
81.71
35.19
31.32
GPT4.1-mini
77.36
34.57
32.56
Qwen2.5-3B-Instruct
25.43
3.72
2.17
Qwen3-4B-Thinking-2507
7.44
1.71
0.31
Qwen2.5-7B-Instruct
43.88
3.72
1.71
Table 3. Benchmarking results of various models on VIRAASAT dataset across different settings (Zero-shot inference, SFT on CoT data, SFT on SCoM data) on the test set. Reported metrics are State Match %, Answer Match % and Full Match % respectively. The best results per column are highlighted in bold.
Hyperparameter
Value
max_new_tokens
512
temperature
0.0
learning_rate
5×10−5
optimizer
paged_adamw_8bit
gradient_accumulation_steps
4
lora_alpha
32
Table 4. Training Hyperparameters (SFT CoT)
Model (Qwen2.5-7B-Instruct)
State match (%)
Answer match (%)
Full Match (%)
SCoM (all components)
88
52
52
w/o critic
75
37
36
w/o manipulations
67
12
9
Table 5. Ablation results for SCoM showing the impact of critic agent and atomic manipulations respectively. Trained on 1k samples and evaluated on 100 test samples.
Model
Attribute-level Accuracy (%)
Qwen2.5-3B-Instruct
75.57
Qwen2.5-7B-Instruct
80.84
Qwen2.5-3B-Instruct (SCoM)
79.25
Qwen2.5-7B-Instruct (SCoM)
84.97
Table 6. OOD Evaluation Results on 10 % SANSKRITI subset. Attribute-level accuracy % is reported.
Figure 6. Sample error analysis for SFT (CoT). The model infers the correct state entity but hallucinates the final answer artifact.
Figure 7. Sample error analysis for SFT (SCoM). The model accurately answers the bridge state and final answer artifact.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 8. Prompt used for the SCoM Actor agent to generate structured reasoning traces under a strict THOUGHT/ACTION/OBSERVATION protocol.
Existing research largely reduces cultural intelligence in LLMs to a knowledge-level problem, overlooking whether models can effectively utilize their acquired knowledge in realistic scenarios. To bridge this gap, we introduce CultureForest, a benchmark for Cultural Norm Grounded Reasoning. Each question is grounded in a small set of atomic norms, enabling verifiable and attributable evaluation. CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions, and supports a progressive evaluation from multiple-choice to open-ended generation. Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings, accompanied by pronounced cross-region disparities. Through targeted analysis, we uncover several consistent patterns: (1) test-time reasoning yields limited gains and may exacerbate inequity; (2) models exhibit highly shared regional preference structures; (3) model responses are markedly conservative, especially under stricter cultural constraints; and (4) by disentangling cultural knowledge acquisition from cultural reasoning, we show that while LLMs possess substantial cultural knowledge, their performance is further bottlenecked by its effective use. These findings point to a necessary shift from knowledge-centric evaluation toward measuring knowledge-grounded reasoning.
Yangfan Ye, Xiaocheng Feng, Jialong Tang +5
Harbin Institute of Technology · Harvard University · The University of Hong Kong
India is a vast nation of over 1.4 billion people, varied by hundreds of diverse and locally specific traditions and cultures and 22 officially recognized languages. Large language models (LLMs) are now being deployed on a massive scale throughout the mainland as well as in remote villages. However, the common benchmarks - MMLU, BIG-Bench, and TruthfulQA are almost exclusively English- and Western-centric. They do not identify those safety, fairness, and accuracy failures unique to the Indian context. That is the gap Inspect India Evals seeks to fill. It is an open-source framework built on top of UK AISI's Inspect AI platform. It has six benchmarks: Multilingual MMLU across sixteen Indian languages, BharatBBQ (our adaptation of BBQ for Indian social bias), a safety evaluation for Digital Public Infrastructure, a multilingual safety test using harmful prompts in Indian languages, a multi-turn jailbreak resistance test, and an Indian cultural knowledge benchmark scored using LLM-as-judge rubrics. In this study, we tested five open-weight models ranging from 8B to 32B parameters. Sarvam-M 24B and Gemma 2 27B came out on top, both scoring 80% on the composite India Fairness Index, with Sarvam-M even beating larger 32B models on Indian cultural knowledge and DPI safety compliance. All models scored 100% refusal on Multilingual Safety, whereas DPI safety varied from 20% to 100%. The framework is public. It's built to work with the UK AISI registry. Anyone can reproduce or extend this work.
As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.
My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan +161