VIRAASAT: Traversing Novel Paths for Indian Cultural Reasoning

Harshul Raj Surana; Arijit Maji; Aryan Vats; Akash Ghosh; Sriparna Saha; Amit Sheth

cs.CLFeb 20, 2026

VIRAASAT: Traversing Novel Paths for Indian Cultural Reasoning

Authors: Harshul Raj Surana, Arijit Maji, Aryan Vats, Akash Ghosh, Sriparna Saha, Amit Sheth

Organizations: AI Institute, University of South Carolina, USA · Indian Institute of Technology Patna, India

Abstract

Large Language Models (LLMs) have made significant progress in reasoning tasks across various domains such as mathematics and coding. However, their performance deteriorates in tasks requiring rich socio-cultural knowledge and diverse local contexts, particularly those involving Indian Culture. Existing Cultural benchmarks are (i) Manually crafted, (ii) contain single-hop questions testing factual recall, and (iii) prohibitively costly to scale, leaving this deficiency largely unmeasured. To address this, we introduce VIRAASAT, a novel, semi-automated multi-hop approach for generating cultural specific multi-hop Question-Answering dataset for Indian culture. VIRAASAT leverages a Knowledge Graph comprising more than 700 expert-curated cultural artifacts, covering 13 key attributes of Indian culture (history, festivals, etc). VIRAASAT spans all 28 states and 8 Union Territories, yielding more than 3,200 multi-hop questions that necessitate chained cultural reasoning. We evaluate current State-of-the-Art (SOTA) LLMs on VIRAASAT and identify key limitations in reasoning wherein fine-tuning on Chain-of-Thought(CoT) traces fails to ground and synthesize low-probability facts. To bridge this gap, we propose a novel framework named Symbolic Chain-of-Manipulation (SCoM). Adapting the Chain-of-Manipulation paradigm, we train the model to simulate atomic Knowledge Graph manipulations internally. SCoM teaches the model to reliably traverse the topological structure of the graph. Experiments on Supervised Fine-Tuning (SFT) demonstrate that SCoM outperforms standard CoT baselines by up to 20%. We release the VIRAASAT dataset along with our findings, laying a strong foundation towards building Culturally Aware Reasoning Models.

Figures & tables

Figure 1. VIRAASAT dataset construction pipeline: artifact curation and identifier creation, Knowledge Graph construction, template-based 2-hop question generation, and expert verification for semantic and grammatical quality.

Panel of Figure 2 — Figure 2. State-wise and attribute-wise distribution of questions in VIRAASAT .

Question	Graph Path (Anchor $\rightarrow$ State $\rightarrow$ Target)	Answer	Chain-of-Thought Trace
Which dish, a savoury fried bread made primarily from powdered rice, originated in the same state where the festival Baha Parab (the spring festival of the Santhal tribe) is celebrated?	festival:baha_parab $\rightarrow$ state:jharkhand $\rightarrow$ cuisine:dhooska	Dhooska	(1) Baha Parab is celebrated in Jharkhand. (2) Dhooska originated in Jharkhand. (3) Therefore, the dish is Dhooska.
Which dish, famous as a fermented sweet made with milk and sugar or jaggery, originated in the same state where the Garad saree is worn?	costume:garad_saree $\rightarrow$ state:west_bengal $\rightarrow$ cuisine:mishti_doi	Mishti doi	(1) Garad saree is from West Bengal. (2) Mishti doi is from West Bengal. (3) Therefore, the dish is Mishti doi.
Which sacred ghat, located on the banks of the Ganges and considered one of the most important pilgrimage sites, is situated in the same state where the “Ramman festival” is celebrated?	festival:ramman_festival $\rightarrow$ state:uttarakhand $\rightarrow$ tourism:har_ki_pauri	Har ki Pauri	(1) Ramman festival is celebrated in Uttarakhand. (2) Har ki Pauri is in Uttarakhand. (3) Therefore, the ghat is Har ki Pauri.

Table 1. Sample data instances from VIRAASAT dataset. Each question requires traversing a constrained 2-hop path from an anchor entity to a bridge state and then to the target artifact.

Dataset	# Train samples	# Val samples	# Test samples
VIRAASAT	2252	323	643
Total	3218 questions

Table 2. Dataset statistics for VIRAASAT .

Figure 3. SCoM reasoning-trace generation pipeline: The figure illustrates the SCoM reasoning trace generation pipeline. The Student Agent (actor) solves the question by producing an explicit, step-wise trace consisting of THOUGHT and ACTION steps, where each ACTION represents an atomic manipulation over the cultural knowledge graph (entity grounding, enforcing the state constraint, candidate retrieval and resolution, and target resolution). Each proposed action is checked by the Symbolic Verifier (teacher) that validates graph-topology consistency with the constrained path; if an action violates constraints, the verifier injects a correction signal that redirects the next step, preventing drift from the intended traversal. Valid actions are executed against the parametric environment backed by the cultural knowledge graph, returning OBSERVATION outputs such as candidate entities and textual descriptors used to guide subsequent steps. The actor–verifier loop iterates for a bounded number of turns (k=5), producing a faithful, tool-grounded SCoM reasoning trace that is used for downstream SFT, teaching a model to perform structured graph-grounded retrieval and reasoning.

Panel of Figure 4 — Figure 4. Example SCoM trace (cleaned) illustrating tool-grounded retrieval over a constrained 2-hop path.

Panel of Figure 5 — Figure 5. Prompt used for SFT, providing the necessary task context and details.

Zero-shot
Model	State Match (%)	Answer Match (%)	Full Match (%)
gemini-2.0-flash	81.71	35.19	31.32
GPT4.1-mini	77.36	34.57	32.56
Qwen2.5-3B-Instruct	25.43	3.72	2.17
Qwen3-4B-Thinking-2507	7.44	1.71	0.31
Qwen2.5-7B-Instruct	43.88	3.72	1.71

Table 3. Benchmarking results of various models on VIRAASAT dataset across different settings (Zero-shot inference, SFT on CoT data, SFT on SCoM data) on the test set. Reported metrics are State Match %, Answer Match % and Full Match % respectively. The best results per column are highlighted in bold.

Hyperparameter	Value
max_new_tokens	512
temperature	0.0
learning_rate	$5\times 10^{-5}$
optimizer	paged_adamw_8bit
gradient_accumulation_steps	4
lora_alpha	32

Table 4. Training Hyperparameters (SFT CoT)

Model (Qwen2.5-7B-Instruct)	State match (%)	Answer match (%)	Full Match (%)
SCoM (all components)	88	52	52
w/o critic	75	37	36
w/o manipulations	67	12	9

Table 5. Ablation results for SCoM showing the impact of critic agent and atomic manipulations respectively. Trained on 1k samples and evaluated on 100 test samples.

Model	Attribute-level Accuracy (%)
Qwen2.5-3B-Instruct	75.57
Qwen2.5-7B-Instruct	80.84
Qwen2.5-3B-Instruct (SCoM)	79.25
Qwen2.5-7B-Instruct (SCoM)	84.97

Table 6. OOD Evaluation Results on 10 % SANSKRITI subset. Attribute-level accuracy % is reported.

Panel of Figure 6 — Figure 6. Sample error analysis for SFT (CoT). The model infers the correct state entity but hallucinates the final answer artifact.

Panel of Figure 7 — Figure 7. Sample error analysis for SFT (SCoM). The model accurately answers the bridge state and final answer artifact.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8. Prompt used for the SCoM Actor agent to generate structured reasoning traces under a strict THOUGHT/ACTION/OBSERVATION protocol.

Cultural Knowledge in Language Models

Cultural knowledge in language models concerns how systems acquire, access, and apply culture-specific facts, norms, meanings, and practices.

See all 132 papers

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs
Jun 1, 2026Yangfan Ye, Xiaocheng Feng, Jialong Tang +5LLM Evaluation Cultural Bias in Language Models
Harbin Institute of Technology · Harvard University · The University of Hong Kong
Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Jul 28, 2026Abhishek Kumar Singh, Shrey Nag, Sachita +2LLM Safety Benchmarks Multilingual Language Model Evaluation
GTBIT¹ · TIET² · IGDTUW³ +2
CulturalRiddles: A Multicultural Benchmark of Riddles
Oct 21, 2026My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan +161Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models

Jun 1, 2026J/K move · Enter open · S save

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs

Yangfan Ye, Xiaocheng Feng, Jialong Tang, Xiayu Cao+4

Harbin Institute of Technology · Harvard University · The University of Hong Kong

LLM Evaluation Cultural Bias in Language Models Language Model Bias Evaluation Cultural Knowledge in Language Models

Open paper AskPDF arXiv

Existing research largely reduces cultural intelligence in LLMs to a knowledge-level problem, overlooking whether models can effectively utilize their acquired knowledge in realistic scenarios. To bridge this gap, we introduce CultureForest, a benchmark for Cultural Norm Grounded Reasoning. Each question is grounded in a small set of atomic norms, enabling verifiable and attributable evaluation. CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions, and supports a progressive evaluation from multiple-choice to open-ended generation. Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings, accompanied by pronounced cross-region disparities. Through targeted analysis, we uncover several consistent patterns: (1) test-time reasoning yields limited gains and may exacerbate inequity; (2) models exhibit highly shared regional preference structures; (3) model responses are markedly conservative, especially under stricter cultural constraints; and (4) by disentangling cultural knowledge acquisition from cultural reasoning, we show that while LLMs possess substantial cultural knowledge, their performance is further bottlenecked by its effective use. These findings point to a necessary shift from knowledge-centric evaluation toward measuring knowledge-grounded reasoning.

Abstract

Cultural Knowledge in Language Models

Explore similar work

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs

Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context

CulturalRiddles: A Multicultural Benchmark of Riddles