Dhara Bench: An Evidence-Oriented Retrieval Benchmark for Indian Knowledge Systems
Organizations: Department of Heritage Science & Technology, Indian Institute of Technology Hyderabad, Sangareddy, Telangana, India. · HTSR Institute, Bengaluru, Karnataka, India. · Department of Biomedical Engineering, Indian Institute of Technology Hyderabad, Sangareddy, Telangana, India. · Bheri Technology Foundation, ITIC, Indian Institute of Technology Hyderabad, Sangareddy, Telangana, India. ·
Abstract
Dhara Bench is a retrieval benchmark for Indian Knowledge Systems (IKS), with 325 questions in 12 categories and 60,121 text chunks from 15 source groups. The categories were developed based on 4,500 questions submitted to the Dhara application; the evaluation set combines corpus-aligned application queries with expert-contributed questions. The annotations comprise 7,578 question-chunk relevance links involving 6,847 distinct corpus chunks. For 34 composite and multi-hop questions, 401 conjugate groups specify alternative sets of passages that jointly support each question. Chunk identifiers and hierarchical references connect the judgments to their source texts. The paper describes corpus preparation, annotation, and validation, and reports a fixed BM25 baseline implemented with Pyserini with and . Across all questions, BM25 achieves MRR@20 of 0.315, Recall@10 of 0.182, and nDCG@10 of 0.205. On the group-annotated subset, complete-evidence scores are CE_Recall@20 = 0.294 and CE_MRR@20 = 0.141. The category results expose particularly weak verse retrieval, while group scores reveal gaps between finding individual passages and completing their support. The corpus and evaluation code are available at https://github.com/Avinash2236/Dhara-Bench.
Figures & tables
| Resource | Primary objective | Knowledge scope | Evidence / grading unit | Relation to Dhārā Bench |
|---|---|---|---|---|
| BEIR ( 20 ) | Information retrieval | Diverse domains; no dedicated IKS collection | Dataset-specific relevance judgments | General heterogeneous IR benchmark |
| KILT ( 13 ) | Knowledge-intensive NLP | Wikipedia | Provenance and task output | Provenance-aware multi-task benchmark |
| Anveshana ( 3 ) | Cross-lingual retrieval | Śrīmadbhāgavatam | English query-Sanskrit document pairs | Retrieval within one IKS work |
| VedantaNY-10M ( 9 ) | RAG / long-form QA | Advaita Vedānta | Retrieved datastore passages | Specialized philosophy retrieval setting |
| Indic-QA ( 18 ) | Context-grounded QA | 11 Indian languages | Supplied QA context | Multilingual QA |
| SANSKRITI ( 8 ) | Cultural knowledge | Indian culture | QA-pair answer evaluation | Cultural knowledge evaluation |
| No. | Source group | IKS domain |
|---|---|---|
| 1 | Mahābhārata | Itihāsa |
| 2 | Vālmīki Rāmāyaṇa | Itihāsa |
| 3 | Agni Purāṇa | Purāṇa |
| 4 | Āpastamba Dharma-sūtra | Dharmaśāstra |
| 5 | Caraka Saṃhitā | Ayurveda |
| 6 | Kathāsaritsāgara | Kāvya |
| Code | Question category | Task description | |
|---|---|---|---|
| T01 | Fact retrieval | Explicitly stated information | 51 |
| T02 | List / set retrieval | Entities satisfying a condition | 40 |
| T03 | Counting / aggregation | Counts, totals, or numerical aggregation | 22 |
| T04 | Chronology / sequence | Order of events or related entities | 25 |
| T05 | Lexical understanding | Meaning, translation, or equivalence | 31 |
| T06 | Composite | Two or more independent questions grouped in one query | 19 |
| Statistic | Value |
|---|---|
| Corpus chunks | 60,121 |
| Unique corpus IDs | 60,121 |
| Total Unicode characters | 50,655,188 |
| Whitespace-delimited words | 8,450,776 |
| Median chunk length (characters) | 615 |
| 90th-percentile chunk length | 1,723 |
| Category | Median | Mean | Max | |
|---|---|---|---|---|
| T01 | 51 | 9.0 | 12.53 | 49 |
| T02 | 40 | 11.0 | 11.93 | 41 |
| T03 | 22 | 9.0 | 11.82 | 64 |
| T04 | 25 | 10.0 | 24.92 | 144 |
| T05 | 31 | 8.0 | 9.48 | 29 |
| T06 | 19 | 4.0 | 7.16 | 31 |
| Metric | Macro score |
|---|---|
| MRR@20 | 0.315 |
| Precision@10 | 0.115 |
| Recall@10 | 0.182 |
| F1@10 | 0.107 |
| nDCG@10 | 0.205 |
| Recall@20 | 0.229 |
| P@ | R@ | F1@ | nDCG@ | |
|---|---|---|---|---|
| 3 | 0.189 | 0.116 | 0.107 | 0.225 |
| 8 | 0.128 | 0.168 | 0.110 | 0.207 |
| 10 | 0.115 | 0.182 | 0.107 | 0.205 |
| 20 | 0.083 | 0.229 | 0.098 | 0.210 |
| Category | MRR@20 | P@3 | P@10 | R@10 | F1@10 | nDCG@10 | R@20 | |
|---|---|---|---|---|---|---|---|---|
| T01 | 51 | 0.455 | 0.255 | 0.159 | 0.188 | 0.133 | 0.262 | 0.248 |
| T02 | 40 | 0.261 | 0.133 | 0.095 | 0.172 | 0.093 | 0.174 | 0.209 |
| T03 | 22 | 0.278 | 0.136 | 0.105 | 0.220 | 0.107 | 0.186 | 0.236 |
| T04 | 25 | 0.376 | 0.240 | 0.140 | 0.172 | 0.112 | 0.223 | 0.192 |
| T05 | 31 | 0.299 | 0.183 | 0.116 | 0.151 | 0.106 | 0.193 | 0.200 |
| T06 | 19 | 0.234 | 0.175 | 0.084 | 0.185 | 0.086 | 0.172 | 0.245 |
| CE_Recall@ | CE_MRR@ | ||||||
|---|---|---|---|---|---|---|---|
| Scope | |||||||
| T06 | 19 | 0.211 | 0.211 | 0.211 | 0.316 | 0.132 | 0.132 |
| T07 | 15 | 0.133 | 0.200 | 0.200 | 0.267 | 0.153 | 0.158 |
| T06+T07 | 34 | 0.176 | 0.206 | 0.206 | 0.294 | 0.141 | 0.144 |
Evidence Retrieval
Evidence retrieval finds and organizes source-grounded information to answer questions, verify claims, and support decisions.
See all 168 papers