cs.CLApr 15, 2026

IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages

Authors: Aviral Dawar, Roshan Karanth, Vikram Goyal, Dhruv Kumar

Organizations: Independent Researcher · BITS Pilani, Pilani, India · IIIT Delhi, New Delhi, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While Large Language Models (LLMs) have significantly advanced Text-to-SQL performance, existing benchmarks predominantly focus on Western contexts and simplified schemas, leaving a gap in real-world, non-Western applications. We present IndicDB, a multilingual Text-to-SQL benchmark for evaluating cross-lingual semantic parsing across diverse Indic languages. The relational schemas are sourced from open-data platforms, including the National Data and Analytics Platform (NDAP) and the India Data Portal (IDP), ensuring realistic administrative data complexity. IndicDB comprises 20 databases across 237 tables. To convert denormalized government data into rich relational structures, we employ an iterative three-agent framework (Architect, Auditor, Refiner) to ensure structural rigor and high relational density (11.85 tables per database; join depths up to six). Our pipeline is value-aware, difficulty-calibrated, and join-enforced, generating 15,617 tasks across English, Hindi, and five Indic languages. We evaluate cross-lingual semantic parsing performance of state-of-the-art models (DeepSeek v3.2, MiniMax 2.7, LLaMA 3.3, Qwen3) across seven linguistic variants. Results show a 9.00% performance drop from English to Indic languages, revealing an "Indic Gap" driven by harder schema linking, increased structural ambiguity, and limited external knowledge. IndicDB serves as a rigorous benchmark for multilingual Text-to-SQL. Code and data: https://anonymous.4open.science/r/multilingualText2Sql-Indic--DDCC/

Figures & tables

Figure 1
Feature / ConstraintEasyMediumHard
Relational Depth0–1 JOINExactly 1 JOIN≥\geq 2 JOINs
JOIN DiversityINNER JOIN  
onlyINNER JOIN  
primarilyDiverse (INNER,  
LEFT, RIGHT)   
Filtering LogicSimple  
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTable 1: IndicDB Comprehensive Framework: Qualitative Complexity Taxonomy (Left) and Full Quantitative Benchmark Statistics (Right).
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishFigure 3: Example of a generated multilingual task
Mistranslation ExamplesFix
Q_en: List the names of districts that produced maize but did not produce any wheat during the year 1970, sorted alphabetically. Q_ma (bad): \marathifont १९७० साली ज्वारी पिकवलेले पण गहू पिकवले नसलेल्या जिल्ह्यांची नावे वर्णक्रमानुसार सूचीबद्ध करा. Q_ma (fix): \marathifont १९७० साली मका पिकवलेले पण गहू पिकवले नसलेल्या जिल्ह्यांची नावे वर्णक्रमानुसार सूचीबद्ध करा.Corrected the lexical mistranslation by replacing \marathifont ज्वारी (sorghum) with \marathifont मका (maize). Using the wrong crop alters the query semantics and can lead to incorrect filtering in SQL generation.
Q_en: List the districts in India for the 1991 census year, ordered by the number of male workers in trade and commerce in descending order, and show only the top 10 results. Q_bn (bad): \bengalifont ১৯৯১ সালের আদমশুমারি অনুযায়ী, বাণিজ্য ও ব্যবসায়ে পুরুষ শ্রমিক সংখ্যার উপর ভিত্তি করে উর্ধ্রক্রমে সাজানো ভারতের জেলাগুলির তালিকা দিন এবং শুধুমাত্র শীর্ষ ১০টি ফলাফল দেখান। Q_bn (fix): \bengalifont ১৯৯১ সালের আদমশুমারি অনুযায়ী, বাণিজ্য ও ব্যবসায়ে পুরুষ শ্রমিক সংখ্যার উপর ভিত্তি করে অবরোহ ক্রমে সাজানো ভারতের জেলাগুলির তালিকা দিন এবং শুধুমাত্র শীর্ষ ১০টি ফলাফল দেখান।Corrected the ordering direction by replacing \bengalifont উর্ধ্রক্রমে (ascending order) with \bengalifont অবরোহ ক্রমে (descending order) to match the intended sorting in the query.
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTable 2: Mistranslation examples and corresponding fixes
MethodModelEnglishHindiBengaliMarathiTamilTeluguHinglish
DIN-SQL (w/ evidence)LLaMA 3.3 70B Instruct66.10%57.97%62.21%54.52%58.09%57.98%65.07%
Qwen3 8B55.05%52.65%51.14%49.36%49.38%49.98%51.06%
DeepSeek V3.269.07%64.45%65.36%63.60%66.67%63.61%66.17%
Minimax 2.762.86%59.05%62.73%63.67%68.07%62.00%65.10%
Zero-shot (w/o evidence)LLaMA 3.3 70B Instruct58.06%44.24%45.14%42.20%42.30%39.46%43.13%
Qwen3 8B52.17%38.39%37.52%36.40%34.23%34.50%38.58%
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTable 3: Execution accuracy (EA) across languages for different prompting strategies. The bold values indicate the highest performance for each method and model configuration, while the final rows quantify the performance degradation across the Indic linguistic spectrum.
Figure 6
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishFigure 6: Case studies illustrating lexical and structural errors across languages.
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishFigure 7: COMET-QE quality score distributions: (Left) aggregate distribution across the corpus, (Right) language-specific breakdown detailing the variance used for targeted human audit.
Question – Evidence pairs
QuestionProvide the area, production, and yield statistics for maize and barley in Chhattisgarh for the year 1970.
EvidenceSelect maize and barley area, production, yield from fact_cereals_minor where dim_geography.state_name = ’Chhattisgarh’ and dim_year.year = 1970.
QuestionList the station code and the type of water body for all stations located in the state of Assam.
EvidenceAssam is a value in dim_state.state; join dim_station with dim_state on state_id; select station_code and type_of_water_body.
Appendix
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTable 4: Question–Evidence pairs for Text-to-SQL reasoning
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishFigure 8: Impact of evidence files: (Left) distribution of improvements, (Right) execution accuracy gains across languages.
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishFigure 9: Schema diagram for a generated schema
More in this topic

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

Explore similar work

CardsList
Jun 6, 2026

UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL

Existing text-to-SQL benchmarks are largely centered on SQLite, making it difficult to evaluate whether models can generalize across heterogeneous SQL dialects. However, real-world database systems differ substantially in syntax, functions, type systems, and execution semantics, so the same natural language intent often requires dialect-specific SQL realizations. We introduce UniQL, a human-verified benchmark for cross-dialect text-to-SQL evaluation. UniQL aligns 1,534 natural language questions with executable SQL annotations across 16 SQL dialects, yielding 24,544 dialect-specific queries. All dialects share the same intents, aligned schemas and database contents, enabling controlled evaluation of dialect generalization. UniQL is constructed through a hybrid pipeline combining database migration, SQL translation, execution-guided verification, iterative rule summarization, and human validation. Experiments on both open-source and closed-source LLMs show that current models remain far from dialect-universal, with substantial performance variation across database systems and limited transfer from SQLite success to other dialects. These findings highlight the need for aligned cross-dialect benchmarks and more dialect-aware text-to-SQL methods. Code and data are available at https://github.com/JerryGao818/UniQL

Jianling Gao, Chongyang Tao, Jiayuan Bai +7

1SKLCCSE, Beihang University · 2The University of Hong Kong

LLM EvaluationText-to-SQLRelational DatabasesText-to-SQL Evaluation
AskPDF
Sep 27, 2026

Closing the Cross-Dialect Gap: Query Plans as a Portable Interface in Text-to-SQL

Text-to-SQL systems are typically trained and evaluated on a single dialect (SQLite), yet production deployments span PostgreSQL, MySQL, ClickHouse, and beyond. We show that this single-dialect assumption leads to a substantial drop in cross-dialect accuracy for every model we tested. The drop persists across scale, architecture, and even purpose-built text-to-SQL systems. We argue that the fix is to change the generation target: instead of asking an LLM to emit dialect-specific SQL, we have it emit a dialect-agnostic relational algebra query plan, which a deterministic compiler then renders into SQL for any supported backend. Across thirteen models from 3B to frontier scale, this restores cross-dialect portability nearly uniformly, at a small cost in peak accuracy on the model's home dialect for capable prompted models and none once fine-tuned on plans; under matched fine-tuning, plan supervision yields a stronger model than SQL supervision. We also introduce MetricName, a question-aware result-set comparator needed to evaluate fairly across dialects, where existing metrics confound semantic errors with benign cross-dialect variation. More broadly, the result is a reminder that a generation target chosen for execution is not necessarily the one that maximizes generation quality.

Corentin Royer, Robin Oester, Yotam Perlitz +3

IBM Research, Zurich, Switzerland · ETH Zurich, Zurich, Switzerland

Text-to-SQLRelational DatabasesText-to-SQL EvaluationDatabase Systems
AskPDF
Aug 16, 2026

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.

Rinit Jain, Tirthraj Mahajan, Advait Joshi +1

Pune Institute of Computer Technology, Pune · L3Cube Labs, Pune · Indian Institute of Technology Madras

Multilingual QAFactual Knowledge in Language ModelsMultilingual Language Model EvaluationCultural Knowledge in Language Models
AskPDF