Hundreds of millions of Indians communicate with digital systems in romanized Latin characters -- typing "aap kaise hain" rather than native script -- yet romanized Indic generation is absent from NLP benchmarks and evaluation frameworks. We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation. We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs. On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004. Two distinct model-specific failure modes emerge: Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses). We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses -- revealing failures metrics alone cannot detect. Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.
Figures & tables
Model
Lang.
BGE
BERT
Inv.
Claude Opus-4.8
Hindi
0.837
0.676
0
Telugu
0.789
0.671
4
Kannada
0.733
0.620
5
GPT-5.5
Hindi
0.828
0.663
0
Telugu
0.794
0.667
0
Kannada
0.749
0.641
2
Table 1: Failure-aware automatic evaluation scores (response only). BGE is BGE-M3 cosine similarity; BERT is BERTScore F1 using bert-base-multilingual-cased . Inv. counts rows receiving zero (empty, English-only, script corruption, repetition) out of 142 total.
Figure 1: BGE-M3 cosine similarity for romanized queries (left) and responses (right) across all four models and three languages. Hindi shows highest scores (0.763–0.837), with Telugu and Kannada showing lower scores due to reduced training data availability.
Figure 2: Multilingual BERTScore (F1, Precision, Recall) for romanized queries (top row) and responses (bottom row) across all four models and three languages. Scores use bert-base-multilingual-cased and are failure-aware (invalid rows receive zero).
Model
Lang.
Eng.-only
Script sp.
Claude 4.8
Hindi
0
0
Telugu
0
3
Kannada
0
5
GPT-5.5
Hindi
0
0
Telugu
0
0
Kannada
0
2
Table 2: Failure breakdown per model and language (out of 142). Eng.-only: response returned in English (word-overlap =1.0 , strict match to source) instead of the romanized target. Script sp. (spillage): response contains one or more non-Latin characters (native script, Cyrillic, etc.), after excluding legitimate romanization diacritics (ā, ē, ṭ). A broader threshold (word-overlap ≥0.90 ) yields 70 English-only cases across all models; the strict =1.0 threshold used here counts only verbatim copies (54 cases).
Model
Nat.
Gram.
Word
Reg.
Usab.
Total
BGE-M3
BERT-F1
Eng.-only
/3
/2
/2
/2
/3
/12
(mean)
(mean)
fails
GPT-5.5
2.49
1.93
1.97
1.94
2.40
10.73
0.790
0.657
0
Claude Opus 4.8
2.05
1.66
1.70
1.73
2.15
9.30
0.786
0.656
2
Gemini 2.5-flash
1.89
1.73
1.54
1.70
2.06
8.91
0.686
0.578
2
Qwen3.5-35B
0.76
0.89
0.93
1.04
0.85
4.48
0.641
0.587
41
Table 3: Human evaluation with automatic-metric summary. Per-dimension human scores are native-speaker judgments (weighted mean over 72 rows, 24 per language): Nat.=Naturalness/Fluency, Gram.=Grammar, Word=Word Choice, Reg.=Register, Usab.=Usability; Total is their sum (max 12). BGE-M3 and BERT-F1 are failure-aware means over all 142 responses per model. Eng.-only fails: genuine cases (excluding English-task prompts) where the model returned English instead of the romanized target.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Hindi
Telugu
Kannada
Overall
Claude 4.8
1,059
1,037
1,107
1,068
GPT-5.5
714
691
737
714
Gemini-2.5
633
686
717
678
Qwen-3.5
714
447
541
567
Appendix
Table 4: Mean output tokens per model and language. Qwen Kannada excludes 49 degenerate rows; including them raises the Kannada mean to 1,768.
Dimension
Description
Points
1. Response Completeness
Is the response non-empty and reasonably thorough?
0–2
2. Script Purity
Are all characters Latin? No native script leakage?
0–2
3. English-Romanized Balance
Is code-mixing appropriate? (Some is natural, too much sounds foreign)
0–3
4. Romanization Consistency
Are word-to-romanization mappings consistent? No ambiguity?
0–3
5. Semantic Faithfulness
Does meaning carry from English through romanization?
0–3
6. Naturalness & Fluency
Does it read naturally or sound like word-by-word translation?
NLP2CT Lab, Faculty of Information Science and Computing, University of Macau · Department of Applied Computer Science, SRH University, Germany · Populus Group, Pittsburgh, United States of America