bharatSetu: a 1M x 22 bridge for indic translation
Abstract
India's Eighth Schedule recognises 22 languages, yet parallel data is uneven across them, and many language pairs, especially among lower-resource languages such as Bodo, Dogri, Santali and Kashmiri, lack direct bitext. We present BharatSetu 22-Indic 1M, a row-aligned table of 984,764 English sentences, each paired with machine translations into all 22 scheduled languages (21,664,808 Indic cells), generated with the sarvamai/sarvam-translate model family. Because every Indic cell in a row translates the same English sentence, any two columns form a translation pair: 462 directed Indic-to-Indic pairs, or 506 including English, without direct bitext. We call these pivot-derived pairs and propose a confidence layer that lets users choose how strictly each pair is connected. The corpus is machine-generated and not human-verified; quality is expected to vary by language, domain and sentence style, and pivot-derived pairs can inherit errors from both sides. We therefore release BharatSetu as a machine layer and propose a human-correction layer keyed to the same row identifiers, so that each corrected column upgrades every pair that uses it, together with an evaluation protocol. We report no translation-quality results.
Figures & tables
| Column | Description |
|---|---|
| id | Identifier aligned to the original source-corpus row. |
| english | English source sentence. |
| source | Provenance label from the input corpus. |
| Target language columns | One machine-generated translation column per target language, named by the full language name. |
| Language | Code | Layer | Language | Code | Layer |
|---|---|---|---|---|---|
| Assamese | as | machine | Bengali | bn | machine |
| Bodo | brx | machine | Dogri | doi | machine |
| Gujarati | gu | machine | Hindi | hi | machine |
| Kannada | kn | machine | Kashmiri | ks | machine |
| Konkani | kok | machine | Maithili | mai | machine |
| Malayalam | ml | machine | Manipuri | mni | machine |
| Evaluation target | Directions | Data condition | Metrics or judgments |
|---|---|---|---|
| Machine-layer MT quality | English 22 and 22 English | Released machine layer; fixed sample per language | chrF, BLEU, COMET where references exist; human adequacy and fluency |
| Pivot-derived pairs | Selected low-resource and high-resource non-English pairs | Pairs derived from shared English rows | Human adequacy; error-propagation categories |
| Confidence layer | Same non-English pairs | Tier 0, Tier 1 and Tier 2 subsets of one pair | Metric deltas on a fixed human-checked test set |
| Correction benefit | Priority languages and sampled rows | Machine layer vs. human-corrected subset | Metric deltas and paired human preference |
| Robustness | Domain, length and script buckets | Stratified samples using source , length and script labels | Error rates by bucket; qualitative audit |
| Responsible-use audit | All languages, sampled | Released and corrected variants | Bias, toxicity, sensitive named-entity and dialect checks |
Low-Resource MT
Low-resource machine translation develops translation systems for languages with scarce parallel text, tools, or standardized evaluation.
See all 85 papers