Retrieval-Augmented Generation (RAG) models have become almost a cornerstone of open-domain question answering tasks; in which the quality of answers generated in the process, greatly relies on the effectiveness of retrieval mechanisms. In the realm of LLM, the last two years have seen tremendous progress in RAG architectures, with many models reporting to have improved performance through various design innovations. The cultural question and answering systems of South Asia, in particular, present a specific challenge since the region is home to the vast number of languages and traditions, which are closely interrelated. Along with these deeply intertwined historical narratives increase the need for culturally grounded and context-aware interpretation. Despite major architectural advances in LLMs, claiming significant improvements, challenges remain when answering these region specific questions in native languages. In this study, we aim to extract high-performing factors to guide the development of optimized hybrid RAG systems. We further explore whether the integration of RAG changes performance differently for culturally grounded and non-cultural questions in a South Asian multilingual setup, and whether RAG produces a higher overall percentage of correct answers than contexted answering. We also examine how LLM performance changes when RAG methods are integrated into the question-answering pipeline in comparison with non-contexted and contexted answering and, evaluate whether the introduction of RAG benefits popular or non-popular culturally grounded questions more. Not only does our work evaluate the current methods but also carves a path for future designs of more accurate and efficient question-answering pipelines.
Figures & tables
Fig. 1: Workflow of the RAG model evaluation framework.
Language
Direct webscraping
After preprocessing
Bangla
17,794
10,769
Hindi
16,362
9,541
Tamil
13,258
11,310
Urdu
8,795
7,805
Nepali
2,641
2,574
TABLE I: Dataset size before and after preprocessing
Language
Final data count
Bangla
10,278
Hindi
8,694
Tamil
8,553
Urdu
5,806
Nepali
2,405
TABLE III: Amount of data present in the final dataset
Title
Context
Question
Cluster
\bengalifont দৈনিক ইত্তেফাক The Daily Ittefaq
\bengalifont দৈনিক ইত্তেফাক বাংলাদেশের বাংলা ভাষার একটি জাতীয় দৈনিক সংবাদপত্র। এটি বাংলা ভাষার প্রিন্ট সংস্করণ ছাড়াও ইংরেজি অনলাইন সংস্করণে প্রকাশিত হয়। The Daily Ittefaq is a national daily newspaper published in the Bengali language in Bangladesh. It also has an English online edition.
\bengalifont বাংলাদেশের বাংলা ভাষার একটি জাতীয় দৈনিক সংবাদপত্র কী? What is a national daily newspaper in the Bengali language of Bangladesh?
organization
\devanagarifont मुज़फ़्फ़रनगर Muzaffarnagar
\devanagarifont मुज़फ़्फ़रनगर भारत के उत्तर प्रदेश राज्य के मुज़फ़्फ़रनगर ज़िले में स्थित एक नगर है। मुज़फ़्फ़रनगर यह ज़िले का मुख्यालय भी है। Muzaffarnagar is a city in Uttar Pradesh, India, and the headquarters of Muzaffarnagar district.
\devanagarifont उत्तर प्रदेश के किस जिले का मुख्यालय उसी नाम के शहर में स्थित है? Which district of Uttar Pradesh has its headquarters in a city with the same name?
locations
\urdufont کشمیری زبان Kashmiri language
\urdufont کشمیری زبان بھارت اور پاکستان کی ایک اہم زبان ہے۔ متکلمین کی تعداد 15 لاکھ سے کچھ اوپر ہے۔ The Kashmiri language is an important language of India and Pakistan, with slightly over 1.5 million speakers.
\urdufont بھارت اور پاکستان میں بولی جانے والی اس زبان کا نام کیا ہے جس کے تقریباً 15 لاکھ سے زائد بولنے والے ہیں؟ What language is spoken in India and Pakistan by over 1.5 million people?
language
\tamilfont பல்லாங்குழி Pallankuzhi
\tamilfont பல்லாங்குழி என்பது, பொதுவாக பெண்கள் விளையாடும் ஒரு விளையாட்டு. முதற் பூப்படைந்த பெண்ணின் தீட்டுக்குரிய காலத்திலும் கருவுற்ற பெண்கள் அமர்ந்து பொழுதுபோக்குவதற்கும் இந்த விளையாட்டை இப்பொழுது ஆடிப் பார்க்கிறார்கள். Pallanguzhi is a traditional game generally played by women.
\tamilfont பெண்கள் விளையாடும் பழமைவாய்ந்த விளையாட்டு எது? Which traditional game is played by women?
entertainment
\devanagarifont हिमालय Himalayas
\devanagarifont हिमालय एसिया को एक हिमशृङ्खला हो, जसले भारतीय उपमहाद्वीप को मैदानलाई तिब्बती पठार बाट अलग गर्छ। यस हिमशृङ्खलामा पृथ्वी का केही अग्लो चुचुराहरू छन्, जसमा सबैभन्दा अग्लो, सगरमाथा पनि समावेश छ। The Himalayas separate the Indian subcontinent from the Tibetan Plateau and include Mount Everest.
\devanagarifont भारतीय उपमहाद्वीपको मैदानलाई तिब्बती पठारबाट अलग पार्ने एवं सगरमाथाले समेटिएको एसियाको एउटा हिमशृङ्खला के हो? What mountain range separates the Indian subcontinent from the Tibetan Plateau and includes Mount Everest?
locations
TABLE IV: Sample of multilingual QA dataset
Lang.
LLM
LC
LG
LI
Bangla
DeepSeek V3
64.0
64.0
57.0
Gemma 3
67.0
66.0
70.0
LLaMA 4
66.0
66.0
68.0
Grok 4.1
61.0
65.0
60.0
Hindi
DeepSeek V3
37.0
38.0
34.0
Gemma 3
46.0
46.0
44.0
TABLE V: Performance comparison across languages and frameworks
Fig. 2: Overall accuracy comparison across non-contexted, contexted, and best RAG settings for five languages. Model groups within each language represent Meta Llama 4, Google Gemma 3, DeepSeek V3-0324, and Grok 4.1 Fast, respectively.
Fig. 3: Accuracy comparison across cultural and non-cultural question categories under contexted and best RAG settings for five languages. Model groups within each language represent Meta Llama 4, Google Gemma 3, DeepSeek V3-0324, and Grok 4.1 Fast, respectively.
Fig. 4: Accuracy comparison for popular titles across cultural and non-cultural question categories under contexted and best RAG settings. Model groups within each language represent Meta Llama 4, Google Gemma 3, DeepSeek V3-0324, and Grok 4.1 Fast, respectively.
Fig. 5: Accuracy comparison for non-popular titles across cultural and non-cultural question categories under contexted and best RAG settings. Model groups within each language represent Meta Llama 4, Google Gemma 3, DeepSeek V3-0324, and Grok 4.1 Fast, respectively.
Fig. 6: Percentage-point improvement of best RAG performance over contexted answering for culturally grounded questions across five languages and four models.
Fig. 7: Percentage-point improvement of best RAG performance over contexted answering for non-cultural questions across five languages and four models.
Fig. 8: Overall accuracy comparison between contexted answering and best RAG performance across five languages.
Fig. 9: Average accuracy across five languages for non-contexted answering, contexted answering, LangGraph RAG, and LlamaIndex RAG.
Fig. 10: RAG gain over non-contexted answering for popular and non-popular culturally grounded questions across five languages.
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.
Although Large Language Models (LLMs) demonstrate excellent capabilities and performance for general reasoning tasks within the general public domain, they may face challenges with culturally grounded knowledge within languages with limited digital and textual data. In this paper, we investigate culturally grounded multiple-choice question answering with the BLEnD benchmark, which consists of a multilingual corpus of 30 languages and covers various socio-cultural domains, such as cuisine, sports, family, etc. We propose a region-aware hybrid retrieval approach that combines BM25 lexical matching and dense semantic similarity with regional weighting heuristics to improve the relevance of the answer. The retrieved documents are used to construct a structured prompt for the Qwen3-14B quantized model with logit-based deterministic answer selection. The experimental results show improvements to cross-lingual stability with the hybrid retrieval approach over pure parametric inference for culturally grounded question answering. However, there are still notable performance gaps between languages with more and less training data. This shows that the limitations of the retrieval augmentation approach are not entirely overcome by the training data imbalance problem.
Hadi Bayrami Asl Tekanlou, Mahdi Bakhtiyarzadeh, Jafar Razmara
Multilingual retrieval-augmented generation (mRAG) is often implemented within a fixed retrieval space, typically via query or document translation or multilingual embedding vector representations. However, this approach may be inadequate for culturally grounded queries, in which retrieval-condition misalignment may occur. Even strong retrievers and generators may struggle to produce culturally relevant answers when sourcing evidence from inappropriate linguistic or regional contexts. To this end, we introduce CORAL (COntext-aware Retrieval with Agentic Loop, an adaptive retrieval methodology for mRAG that enables iterative refinement of both the retrieval space (corpora) and the retrieval probe (query) based on the quality of the evidence. The overall process includes: (1) selecting corpora, (2) retrieving documents, (3) critiquing evidence for relevance and cultural alignment, and (4) checking sufficiency. If the retrieved documents are insufficient to answer the query correctly, the system (5) reselects corpora and rewrites the query. Across two cultural QA benchmarks, CORAL achieves up to a 3.58%p accuracy improvement on low-resource languages relative to the strongest baselines.
Large language models (LLMs) suffer from a long-tail deficit: culturally specific facts, particularly those concerning underrepresented regions such as Latin America, appear too rarely in pretraining corpora to be reliably memorized. Retrieval-Augmented Generation (RAG) addresses this by grounding generation in external text, but structured alternatives such as Knowledge Graphs (KGs) offer tighter control over what enters the context, along with potential gains in explainability and updatability. We benchmark Graph-RAG against standard RAG on LatamQA, a culturally grounded multiple-choice dataset spanning eight thematic categories. The graphs are built end-to-end from Wikipedia articles with KGGen, a recent open-domain extractor, without manual curation in our main setting. G-Retriever is competitive with RAG and reduces the error of the base LLM by 72% with a standard KG and 78% with a benchmark-aware variant, the gap to RAG narrowing further as the graph is oriented toward task-relevant content. The trained projection transfers zero-shot to Portuguese without target-language fine-tuning, indicating multilingual reach.
Pablo Poulenard, Yannis Karmim, Valentin Barrière
DCC, Universidad de Chile, Santiago, Chile · École Polytechnique, Palaiseau, France · Inria, Almanach, Paris, France +2