IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages
Organizations: Independent Researcher · BITS Pilani, Pilani, India · IIIT Delhi, New Delhi, India
Abstract
While Large Language Models (LLMs) have significantly advanced Text-to-SQL performance, existing benchmarks predominantly focus on Western contexts and simplified schemas, leaving a gap in real-world, non-Western applications. We present IndicDB, a multilingual Text-to-SQL benchmark for evaluating cross-lingual semantic parsing across diverse Indic languages. The relational schemas are sourced from open-data platforms, including the National Data and Analytics Platform (NDAP) and the India Data Portal (IDP), ensuring realistic administrative data complexity. IndicDB comprises 20 databases across 237 tables. To convert denormalized government data into rich relational structures, we employ an iterative three-agent framework (Architect, Auditor, Refiner) to ensure structural rigor and high relational density (11.85 tables per database; join depths up to six). Our pipeline is value-aware, difficulty-calibrated, and join-enforced, generating 15,617 tasks across English, Hindi, and five Indic languages. We evaluate cross-lingual semantic parsing performance of state-of-the-art models (DeepSeek v3.2, MiniMax 2.7, LLaMA 3.3, Qwen3) across seven linguistic variants. Results show a 9.00% performance drop from English to Indic languages, revealing an "Indic Gap" driven by harder schema linking, increased structural ambiguity, and limited external knowledge. IndicDB serves as a rigorous benchmark for multilingual Text-to-SQL. Code and data: https://anonymous.4open.science/r/multilingualText2Sql-Indic--DDCC/
Figures & tables
| Feature / Constraint | Easy | Medium | Hard |
| Relational Depth | 0–1 JOIN | Exactly 1 JOIN | 2 JOINs |
| JOIN Diversity | INNER JOIN | ||
| only | INNER JOIN | ||
| primarily | Diverse (INNER, | ||
| LEFT, RIGHT) | |||
| Filtering Logic | Simple |
| Mistranslation Examples | Fix |
| Q_en: List the names of districts that produced maize but did not produce any wheat during the year 1970, sorted alphabetically. Q_ma (bad): \marathifont १९७० साली ज्वारी पिकवलेले पण गहू पिकवले नसलेल्या जिल्ह्यांची नावे वर्णक्रमानुसार सूचीबद्ध करा. Q_ma (fix): \marathifont १९७० साली मका पिकवलेले पण गहू पिकवले नसलेल्या जिल्ह्यांची नावे वर्णक्रमानुसार सूचीबद्ध करा. | Corrected the lexical mistranslation by replacing \marathifont ज्वारी (sorghum) with \marathifont मका (maize). Using the wrong crop alters the query semantics and can lead to incorrect filtering in SQL generation. |
| Q_en: List the districts in India for the 1991 census year, ordered by the number of male workers in trade and commerce in descending order, and show only the top 10 results. Q_bn (bad): \bengalifont ১৯৯১ সালের আদমশুমারি অনুযায়ী, বাণিজ্য ও ব্যবসায়ে পুরুষ শ্রমিক সংখ্যার উপর ভিত্তি করে উর্ধ্রক্রমে সাজানো ভারতের জেলাগুলির তালিকা দিন এবং শুধুমাত্র শীর্ষ ১০টি ফলাফল দেখান। Q_bn (fix): \bengalifont ১৯৯১ সালের আদমশুমারি অনুযায়ী, বাণিজ্য ও ব্যবসায়ে পুরুষ শ্রমিক সংখ্যার উপর ভিত্তি করে অবরোহ ক্রমে সাজানো ভারতের জেলাগুলির তালিকা দিন এবং শুধুমাত্র শীর্ষ ১০টি ফলাফল দেখান। | Corrected the ordering direction by replacing \bengalifont উর্ধ্রক্রমে (ascending order) with \bengalifont অবরোহ ক্রমে (descending order) to match the intended sorting in the query. |
| Method | Model | English | Hindi | Bengali | Marathi | Tamil | Telugu | Hinglish |
| DIN-SQL (w/ evidence) | LLaMA 3.3 70B Instruct | 66.10% | 57.97% | 62.21% | 54.52% | 58.09% | 57.98% | 65.07% |
| Qwen3 8B | 55.05% | 52.65% | 51.14% | 49.36% | 49.38% | 49.98% | 51.06% | |
| DeepSeek V3.2 | 69.07% | 64.45% | 65.36% | 63.60% | 66.67% | 63.61% | 66.17% | |
| Minimax 2.7 | 62.86% | 59.05% | 62.73% | 63.67% | 68.07% | 62.00% | 65.10% | |
| Zero-shot (w/o evidence) | LLaMA 3.3 70B Instruct | 58.06% | 44.24% | 45.14% | 42.20% | 42.30% | 39.46% | 43.13% |
| Qwen3 8B | 52.17% | 38.39% | 37.52% | 36.40% | 34.23% | 34.50% | 38.58% |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Question – Evidence pairs | |
| Question | Provide the area, production, and yield statistics for maize and barley in Chhattisgarh for the year 1970. |
| Evidence | Select maize and barley area, production, yield from fact_cereals_minor where dim_geography.state_name = ’Chhattisgarh’ and dim_year.year = 1970. |
| Question | List the station code and the type of water body for all stations located in the state of Assam. |
| Evidence | Assam is a value in dim_state.state; join dim_station with dim_state on state_id; select station_code and type_of_water_body. |
Text-to-SQL
Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.
See all 173 papers